Jul 23 00:08:04.930730 ip-10-0-131-121 systemd[1]: Starting Kubernetes Kubelet... Jul 23 00:08:05.398230 ip-10-0-131-121 kubenswrapper[2562]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 23 00:08:05.398230 ip-10-0-131-121 kubenswrapper[2562]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 23 00:08:05.398230 ip-10-0-131-121 kubenswrapper[2562]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 23 00:08:05.398230 ip-10-0-131-121 kubenswrapper[2562]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 23 00:08:05.398230 ip-10-0-131-121 kubenswrapper[2562]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 23 00:08:05.398230 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.376947 2562 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 23 00:08:05.398230 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382490 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 00:08:05.398230 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382509 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 00:08:05.398230 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382513 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 00:08:05.398230 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382516 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 00:08:05.398230 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382519 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 00:08:05.398230 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382524 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 00:08:05.398230 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382527 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 00:08:05.398230 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382529 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 00:08:05.399566 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382534 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 00:08:05.399566 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382539 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 00:08:05.399566 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382542 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 00:08:05.399566 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382544 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 00:08:05.399566 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382549 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 00:08:05.399566 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382553 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 00:08:05.399566 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382556 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 00:08:05.399566 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382559 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 00:08:05.399566 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382568 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 00:08:05.399566 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382571 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 00:08:05.399566 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382574 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 00:08:05.399566 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382576 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 00:08:05.399566 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382580 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 00:08:05.399566 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382582 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 00:08:05.399566 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382586 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 00:08:05.399566 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382589 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 00:08:05.399566 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382592 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 00:08:05.399566 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382595 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 00:08:05.399566 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382598 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 00:08:05.400672 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382601 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 23 00:08:05.400672 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382603 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 00:08:05.400672 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382606 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 00:08:05.400672 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382609 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 00:08:05.400672 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382611 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 00:08:05.400672 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382614 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 00:08:05.400672 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382616 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 00:08:05.400672 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382619 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 00:08:05.400672 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382621 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 00:08:05.400672 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382624 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 00:08:05.400672 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382627 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 00:08:05.400672 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382630 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 00:08:05.400672 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382632 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 00:08:05.400672 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382635 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 00:08:05.400672 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382639 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 00:08:05.400672 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382641 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 00:08:05.400672 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382643 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 00:08:05.400672 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382646 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 00:08:05.400672 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382648 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 00:08:05.400672 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382651 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 00:08:05.401572 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382654 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 00:08:05.401572 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382657 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 00:08:05.401572 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382662 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 00:08:05.401572 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382665 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 00:08:05.401572 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382668 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 00:08:05.401572 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382670 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 00:08:05.401572 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382673 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 00:08:05.401572 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382676 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 00:08:05.401572 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382679 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 00:08:05.401572 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382682 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 00:08:05.401572 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382685 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 00:08:05.401572 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382688 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 00:08:05.401572 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382691 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 00:08:05.401572 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382693 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 00:08:05.401572 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382696 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 00:08:05.401572 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382699 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 00:08:05.401572 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382701 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 00:08:05.401572 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382704 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 00:08:05.401572 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382706 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 00:08:05.402678 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382709 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 00:08:05.402678 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382711 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 00:08:05.402678 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382714 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 00:08:05.402678 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382717 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 00:08:05.402678 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382720 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 00:08:05.402678 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382724 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 00:08:05.402678 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382727 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 00:08:05.402678 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382730 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 00:08:05.402678 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382733 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 00:08:05.402678 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382737 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 00:08:05.402678 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382739 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 00:08:05.402678 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382742 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 00:08:05.402678 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382744 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 00:08:05.402678 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382747 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 00:08:05.402678 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382751 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 00:08:05.402678 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382753 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 00:08:05.402678 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382762 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 00:08:05.402678 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382767 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 00:08:05.402678 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382769 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 00:08:05.402678 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382772 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 00:08:05.403604 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382774 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 00:08:05.403604 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382777 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 00:08:05.403604 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382780 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 00:08:05.403604 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382783 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 00:08:05.403604 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382786 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 00:08:05.403604 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382788 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 00:08:05.403604 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382791 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 00:08:05.403604 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382794 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 00:08:05.403604 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382796 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 00:08:05.403604 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382799 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 00:08:05.403604 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382802 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 00:08:05.403604 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382804 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 00:08:05.403604 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.382807 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 00:08:05.403604 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383633 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 00:08:05.403604 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383641 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 00:08:05.403604 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383644 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 00:08:05.403604 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383647 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 00:08:05.403604 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383651 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 00:08:05.403604 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383654 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 00:08:05.403604 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383658 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 00:08:05.404739 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383661 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 00:08:05.404739 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383664 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 00:08:05.404739 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383667 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 00:08:05.404739 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383670 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 00:08:05.404739 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383674 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 00:08:05.404739 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383677 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 00:08:05.404739 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383680 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 00:08:05.404739 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383683 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 00:08:05.404739 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383685 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 23 00:08:05.404739 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383689 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 00:08:05.404739 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383692 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 00:08:05.404739 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383695 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 00:08:05.404739 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383698 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 00:08:05.404739 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383703 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 00:08:05.404739 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383706 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 00:08:05.404739 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383710 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 00:08:05.404739 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383713 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 00:08:05.404739 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383716 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 00:08:05.404739 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383719 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 00:08:05.404739 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383721 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 00:08:05.405859 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383724 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 00:08:05.405859 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383729 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 00:08:05.405859 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383734 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 00:08:05.405859 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383736 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 00:08:05.405859 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383739 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 00:08:05.405859 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383742 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 00:08:05.405859 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383745 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 00:08:05.405859 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383748 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 00:08:05.405859 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383750 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 00:08:05.405859 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383753 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 00:08:05.405859 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383756 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 00:08:05.405859 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383760 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 00:08:05.405859 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383763 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 00:08:05.405859 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383765 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 00:08:05.405859 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383768 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 00:08:05.405859 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383771 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 00:08:05.405859 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383773 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 00:08:05.405859 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383776 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 00:08:05.405859 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383779 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 00:08:05.406778 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383784 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 00:08:05.406778 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383788 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 00:08:05.406778 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383791 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 00:08:05.406778 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383794 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 00:08:05.406778 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383797 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 00:08:05.406778 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383800 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 00:08:05.406778 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383802 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 00:08:05.406778 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383805 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 00:08:05.406778 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383808 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 00:08:05.406778 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383814 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 00:08:05.406778 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383817 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 00:08:05.406778 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383820 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 00:08:05.406778 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383823 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 00:08:05.406778 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383825 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 00:08:05.406778 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383828 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 00:08:05.406778 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383830 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 00:08:05.406778 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383833 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 00:08:05.406778 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383836 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 00:08:05.406778 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383838 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 00:08:05.407897 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383841 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 00:08:05.407897 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383844 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 00:08:05.407897 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383846 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 00:08:05.407897 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383849 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 00:08:05.407897 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383851 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 00:08:05.407897 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383854 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 00:08:05.407897 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383857 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 00:08:05.407897 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383860 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 00:08:05.407897 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383862 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 00:08:05.407897 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383865 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 00:08:05.407897 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383867 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 00:08:05.407897 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383869 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 00:08:05.407897 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383872 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 00:08:05.407897 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383874 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 00:08:05.407897 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383877 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 00:08:05.407897 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383880 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 00:08:05.407897 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383882 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 00:08:05.407897 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383885 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 00:08:05.407897 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383887 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 00:08:05.408801 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383889 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 00:08:05.408801 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383892 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 00:08:05.408801 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383895 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 00:08:05.408801 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383901 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 00:08:05.408801 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383903 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 00:08:05.408801 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383906 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 00:08:05.408801 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383909 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 00:08:05.408801 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383911 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 00:08:05.408801 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383913 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 00:08:05.408801 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383916 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 00:08:05.408801 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383918 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 00:08:05.408801 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383921 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 00:08:05.408801 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383924 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 00:08:05.408801 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383927 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 00:08:05.408801 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.383929 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 00:08:05.408801 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384375 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 00:08:05.408801 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384381 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 00:08:05.408801 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384384 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 00:08:05.408801 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384387 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 00:08:05.408801 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384390 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 00:08:05.409708 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384393 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 00:08:05.409708 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384396 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 00:08:05.409708 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384398 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 00:08:05.409708 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384401 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 00:08:05.409708 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384405 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 00:08:05.409708 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384409 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 00:08:05.409708 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384412 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 00:08:05.409708 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384415 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 00:08:05.409708 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384418 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 00:08:05.409708 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384421 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 00:08:05.409708 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384424 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 00:08:05.409708 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384426 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 00:08:05.409708 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384429 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 00:08:05.409708 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384432 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 00:08:05.409708 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384436 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 00:08:05.409708 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384441 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 00:08:05.409708 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384444 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 00:08:05.409708 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384447 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 00:08:05.409708 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384450 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 00:08:05.410615 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384453 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 00:08:05.410615 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384456 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 00:08:05.410615 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384459 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 00:08:05.410615 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384461 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 00:08:05.410615 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384464 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 00:08:05.410615 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384467 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 00:08:05.410615 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384470 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 00:08:05.410615 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384472 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 00:08:05.410615 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384475 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 00:08:05.410615 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384478 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 00:08:05.410615 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384482 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 00:08:05.410615 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384485 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 00:08:05.410615 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384487 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 00:08:05.410615 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384490 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 00:08:05.410615 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384493 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 00:08:05.410615 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384497 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 00:08:05.410615 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384500 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 00:08:05.410615 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384502 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 00:08:05.410615 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384505 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 00:08:05.410615 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384507 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 00:08:05.411503 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384510 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 00:08:05.411503 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384512 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 00:08:05.411503 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384515 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 00:08:05.411503 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384517 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 00:08:05.411503 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384520 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 00:08:05.411503 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384522 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 00:08:05.411503 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384525 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 00:08:05.411503 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384529 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 00:08:05.411503 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384533 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 00:08:05.411503 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384536 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 00:08:05.411503 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384539 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 00:08:05.411503 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384542 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 00:08:05.411503 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384545 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 00:08:05.411503 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384548 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 00:08:05.411503 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384550 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 00:08:05.411503 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384553 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 00:08:05.411503 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384556 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 00:08:05.411503 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384559 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 00:08:05.411503 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384561 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 00:08:05.412610 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384564 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 23 00:08:05.412610 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384567 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 00:08:05.412610 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384569 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 00:08:05.412610 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384572 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 00:08:05.412610 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384574 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 00:08:05.412610 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384577 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 00:08:05.412610 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384580 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 00:08:05.412610 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384584 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 00:08:05.412610 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384588 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 00:08:05.412610 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384591 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 00:08:05.412610 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384593 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 00:08:05.412610 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384596 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 00:08:05.412610 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384598 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 00:08:05.412610 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384601 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 00:08:05.412610 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384603 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 00:08:05.412610 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384606 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 00:08:05.412610 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384609 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 00:08:05.412610 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384611 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 00:08:05.412610 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384614 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 00:08:05.412610 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384616 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 00:08:05.413506 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384618 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 00:08:05.413506 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384624 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 00:08:05.413506 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384626 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 00:08:05.413506 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384629 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 00:08:05.413506 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384632 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 00:08:05.413506 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384634 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 00:08:05.413506 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384636 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 00:08:05.413506 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384639 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 00:08:05.413506 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384642 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 00:08:05.413506 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384644 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 00:08:05.413506 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384647 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 00:08:05.413506 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384649 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 00:08:05.413506 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384651 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 00:08:05.413506 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384654 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 00:08:05.413506 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384656 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 00:08:05.413506 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.384659 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 00:08:05.413506 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384741 2562 flags.go:64] FLAG: --address="0.0.0.0" Jul 23 00:08:05.413506 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384749 2562 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 23 00:08:05.413506 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384755 2562 flags.go:64] FLAG: --anonymous-auth="true" Jul 23 00:08:05.413506 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384760 2562 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 23 00:08:05.413506 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384768 2562 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 23 00:08:05.414401 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384771 2562 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 23 00:08:05.414401 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384776 2562 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 23 00:08:05.414401 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384781 2562 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 23 00:08:05.414401 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384784 2562 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 23 00:08:05.414401 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384787 2562 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 23 00:08:05.414401 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384791 2562 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 23 00:08:05.414401 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384794 2562 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 23 00:08:05.414401 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384797 2562 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 23 00:08:05.414401 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384800 2562 flags.go:64] FLAG: --cgroup-root="" Jul 23 00:08:05.414401 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384803 2562 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 23 00:08:05.414401 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384806 2562 flags.go:64] FLAG: --client-ca-file="" Jul 23 00:08:05.414401 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384809 2562 flags.go:64] FLAG: --cloud-provider="external" Jul 23 00:08:05.414401 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384814 2562 flags.go:64] FLAG: --cluster-dns="[]" Jul 23 00:08:05.414401 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384819 2562 flags.go:64] FLAG: --cluster-domain="" Jul 23 00:08:05.414401 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384822 2562 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 23 00:08:05.414401 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384825 2562 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 23 00:08:05.414401 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384828 2562 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 23 00:08:05.414401 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384831 2562 flags.go:64] FLAG: --container-log-max-files="5" Jul 23 00:08:05.414401 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384836 2562 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 23 00:08:05.414401 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384839 2562 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 23 00:08:05.414401 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384842 2562 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 23 00:08:05.414401 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384846 2562 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 23 00:08:05.414401 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384849 2562 flags.go:64] FLAG: --contention-profiling="false" Jul 23 00:08:05.414401 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384852 2562 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 23 00:08:05.415707 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384855 2562 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 23 00:08:05.415707 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384858 2562 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 23 00:08:05.415707 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384861 2562 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 23 00:08:05.415707 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384865 2562 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 23 00:08:05.415707 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384868 2562 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 23 00:08:05.415707 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384871 2562 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 23 00:08:05.415707 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384874 2562 flags.go:64] FLAG: --enable-load-reader="false" Jul 23 00:08:05.415707 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384878 2562 flags.go:64] FLAG: --enable-server="true" Jul 23 00:08:05.415707 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384882 2562 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 23 00:08:05.415707 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384886 2562 flags.go:64] FLAG: --event-burst="100" Jul 23 00:08:05.415707 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384889 2562 flags.go:64] FLAG: --event-qps="50" Jul 23 00:08:05.415707 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384892 2562 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 23 00:08:05.415707 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384896 2562 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 23 00:08:05.415707 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384899 2562 flags.go:64] FLAG: --eviction-hard="" Jul 23 00:08:05.415707 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384903 2562 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 23 00:08:05.415707 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384906 2562 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 23 00:08:05.415707 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384909 2562 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 23 00:08:05.415707 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384912 2562 flags.go:64] FLAG: --eviction-soft="" Jul 23 00:08:05.415707 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384915 2562 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 23 00:08:05.415707 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384918 2562 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 23 00:08:05.415707 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384922 2562 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 23 00:08:05.415707 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384925 2562 flags.go:64] FLAG: --experimental-mounter-path="" Jul 23 00:08:05.415707 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384928 2562 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 23 00:08:05.415707 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384931 2562 flags.go:64] FLAG: --fail-swap-on="true" Jul 23 00:08:05.415707 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384934 2562 flags.go:64] FLAG: --feature-gates="" Jul 23 00:08:05.416832 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384938 2562 flags.go:64] FLAG: --file-check-frequency="20s" Jul 23 00:08:05.416832 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384942 2562 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 23 00:08:05.416832 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384945 2562 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 23 00:08:05.416832 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384948 2562 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 23 00:08:05.416832 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384951 2562 flags.go:64] FLAG: --healthz-port="10248" Jul 23 00:08:05.416832 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384954 2562 flags.go:64] FLAG: --help="false" Jul 23 00:08:05.416832 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384957 2562 flags.go:64] FLAG: --hostname-override="ip-10-0-131-121.ec2.internal" Jul 23 00:08:05.416832 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384960 2562 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 23 00:08:05.416832 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384963 2562 flags.go:64] FLAG: --http-check-frequency="20s" Jul 23 00:08:05.416832 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384966 2562 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 23 00:08:05.416832 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384969 2562 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 23 00:08:05.416832 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384973 2562 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 23 00:08:05.416832 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384976 2562 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 23 00:08:05.416832 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384978 2562 flags.go:64] FLAG: --image-service-endpoint="" Jul 23 00:08:05.416832 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384982 2562 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 23 00:08:05.416832 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384986 2562 flags.go:64] FLAG: --kube-api-burst="100" Jul 23 00:08:05.416832 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384989 2562 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 23 00:08:05.416832 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384992 2562 flags.go:64] FLAG: --kube-api-qps="50" Jul 23 00:08:05.416832 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384995 2562 flags.go:64] FLAG: --kube-reserved="" Jul 23 00:08:05.416832 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.384998 2562 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 23 00:08:05.416832 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385001 2562 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 23 00:08:05.416832 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385004 2562 flags.go:64] FLAG: --kubelet-cgroups="" Jul 23 00:08:05.416832 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385007 2562 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 23 00:08:05.416832 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385010 2562 flags.go:64] FLAG: --lock-file="" Jul 23 00:08:05.417929 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385013 2562 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 23 00:08:05.417929 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385016 2562 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 23 00:08:05.417929 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385019 2562 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 23 00:08:05.417929 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385026 2562 flags.go:64] FLAG: --log-json-split-stream="false" Jul 23 00:08:05.417929 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385029 2562 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 23 00:08:05.417929 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385032 2562 flags.go:64] FLAG: --log-text-split-stream="false" Jul 23 00:08:05.417929 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385034 2562 flags.go:64] FLAG: --logging-format="text" Jul 23 00:08:05.417929 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385037 2562 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 23 00:08:05.417929 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385041 2562 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 23 00:08:05.417929 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385043 2562 flags.go:64] FLAG: --manifest-url="" Jul 23 00:08:05.417929 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385046 2562 flags.go:64] FLAG: --manifest-url-header="" Jul 23 00:08:05.417929 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385050 2562 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 23 00:08:05.417929 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385054 2562 flags.go:64] FLAG: --max-open-files="1000000" Jul 23 00:08:05.417929 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385058 2562 flags.go:64] FLAG: --max-pods="110" Jul 23 00:08:05.417929 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385061 2562 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 23 00:08:05.417929 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385064 2562 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 23 00:08:05.417929 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385067 2562 flags.go:64] FLAG: --memory-manager-policy="None" Jul 23 00:08:05.417929 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385070 2562 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 23 00:08:05.417929 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385073 2562 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 23 00:08:05.417929 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385076 2562 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 23 00:08:05.417929 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385079 2562 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 23 00:08:05.417929 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385087 2562 flags.go:64] FLAG: --node-status-max-images="50" Jul 23 00:08:05.417929 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385092 2562 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 23 00:08:05.417929 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385095 2562 flags.go:64] FLAG: --oom-score-adj="-999" Jul 23 00:08:05.417929 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385098 2562 flags.go:64] FLAG: --pod-cidr="" Jul 23 00:08:05.421712 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385111 2562 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 23 00:08:05.421712 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385117 2562 flags.go:64] FLAG: --pod-manifest-path="" Jul 23 00:08:05.421712 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385120 2562 flags.go:64] FLAG: --pod-max-pids="-1" Jul 23 00:08:05.421712 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385123 2562 flags.go:64] FLAG: --pods-per-core="0" Jul 23 00:08:05.421712 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385126 2562 flags.go:64] FLAG: --port="10250" Jul 23 00:08:05.421712 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385129 2562 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 23 00:08:05.421712 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385132 2562 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02d682c5d0ed29f68" Jul 23 00:08:05.421712 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385135 2562 flags.go:64] FLAG: --qos-reserved="" Jul 23 00:08:05.421712 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385138 2562 flags.go:64] FLAG: --read-only-port="10255" Jul 23 00:08:05.421712 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385141 2562 flags.go:64] FLAG: --register-node="true" Jul 23 00:08:05.421712 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385144 2562 flags.go:64] FLAG: --register-with-taints="" Jul 23 00:08:05.421712 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385149 2562 flags.go:64] FLAG: --registry-burst="10" Jul 23 00:08:05.421712 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385152 2562 flags.go:64] FLAG: --registry-qps="5" Jul 23 00:08:05.421712 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385155 2562 flags.go:64] FLAG: --reserved-cpus="" Jul 23 00:08:05.421712 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385158 2562 flags.go:64] FLAG: --reserved-memory="" Jul 23 00:08:05.421712 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385162 2562 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 23 00:08:05.421712 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385165 2562 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 23 00:08:05.421712 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385168 2562 flags.go:64] FLAG: --rotate-certificates="false" Jul 23 00:08:05.421712 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385171 2562 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 23 00:08:05.421712 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385174 2562 flags.go:64] FLAG: --runonce="false" Jul 23 00:08:05.421712 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385177 2562 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 23 00:08:05.421712 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385180 2562 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 23 00:08:05.421712 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385183 2562 flags.go:64] FLAG: --seccomp-default="false" Jul 23 00:08:05.421712 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385186 2562 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 23 00:08:05.421712 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385189 2562 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 23 00:08:05.422833 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385191 2562 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 23 00:08:05.422833 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385195 2562 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 23 00:08:05.422833 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385198 2562 flags.go:64] FLAG: --storage-driver-password="root" Jul 23 00:08:05.422833 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385201 2562 flags.go:64] FLAG: --storage-driver-secure="false" Jul 23 00:08:05.422833 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385206 2562 flags.go:64] FLAG: --storage-driver-table="stats" Jul 23 00:08:05.422833 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385209 2562 flags.go:64] FLAG: --storage-driver-user="root" Jul 23 00:08:05.422833 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385212 2562 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 23 00:08:05.422833 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385215 2562 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 23 00:08:05.422833 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385224 2562 flags.go:64] FLAG: --system-cgroups="" Jul 23 00:08:05.422833 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385227 2562 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 23 00:08:05.422833 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385232 2562 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 23 00:08:05.422833 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385235 2562 flags.go:64] FLAG: --tls-cert-file="" Jul 23 00:08:05.422833 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385238 2562 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 23 00:08:05.422833 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385245 2562 flags.go:64] FLAG: --tls-min-version="" Jul 23 00:08:05.422833 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385248 2562 flags.go:64] FLAG: --tls-private-key-file="" Jul 23 00:08:05.422833 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385250 2562 flags.go:64] FLAG: --topology-manager-policy="none" Jul 23 00:08:05.422833 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385253 2562 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 23 00:08:05.422833 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385256 2562 flags.go:64] FLAG: --topology-manager-scope="container" Jul 23 00:08:05.422833 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385260 2562 flags.go:64] FLAG: --v="2" Jul 23 00:08:05.422833 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385265 2562 flags.go:64] FLAG: --version="false" Jul 23 00:08:05.422833 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385269 2562 flags.go:64] FLAG: --vmodule="" Jul 23 00:08:05.422833 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385273 2562 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 23 00:08:05.422833 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385276 2562 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 23 00:08:05.422833 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385426 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 00:08:05.422833 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385430 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 00:08:06.643569 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385433 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 00:08:06.643569 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385436 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 00:08:06.643569 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385438 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 00:08:06.643569 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385441 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 00:08:06.643569 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385444 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 00:08:06.643569 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385446 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 00:08:06.643569 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385449 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 00:08:06.643569 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385452 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 00:08:06.643569 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385454 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 00:08:06.643569 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385457 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 00:08:06.643569 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385460 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 00:08:06.643569 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385464 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 00:08:06.643569 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385466 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 00:08:06.643569 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385469 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 00:08:06.643569 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385471 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 00:08:06.643569 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385474 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 00:08:06.643569 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385477 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 00:08:06.643569 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385480 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 00:08:06.643569 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385483 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 00:08:06.643569 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385485 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 00:08:06.639819 ip-10-0-131-121 systemd[1]: Started Kubernetes Kubelet. Jul 23 00:08:06.644725 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385488 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 00:08:06.644725 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385491 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 00:08:06.644725 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385493 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 00:08:06.644725 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385496 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 00:08:06.644725 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385498 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 00:08:06.644725 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385502 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 00:08:06.644725 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385505 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 00:08:06.644725 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385507 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 00:08:06.644725 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385512 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 00:08:06.644725 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385515 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 00:08:06.644725 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385518 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 00:08:06.644725 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385521 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 00:08:06.644725 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385524 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 00:08:06.644725 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385526 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 00:08:06.644725 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385529 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 00:08:06.644725 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385532 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 00:08:06.644725 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385535 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 00:08:06.644725 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385537 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 00:08:06.644725 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385540 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 00:08:06.645858 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385542 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 00:08:06.645858 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385544 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 00:08:06.645858 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385547 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 00:08:06.645858 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385549 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 00:08:06.645858 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385553 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 00:08:06.645858 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385556 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 23 00:08:06.645858 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385558 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 00:08:06.645858 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385561 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 00:08:06.645858 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385564 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 00:08:06.645858 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385568 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 00:08:06.645858 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385572 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 00:08:06.645858 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385575 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 00:08:06.645858 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385578 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 00:08:06.645858 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385581 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 00:08:06.645858 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385583 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 00:08:06.645858 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385586 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 00:08:06.645858 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385588 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 00:08:06.645858 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385591 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 00:08:06.645858 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385595 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 00:08:06.645858 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385597 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 00:08:06.646943 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385600 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 00:08:06.646943 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385603 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 00:08:06.646943 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385606 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 00:08:06.646943 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385609 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 00:08:06.646943 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385611 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 00:08:06.646943 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385614 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 00:08:06.646943 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385616 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 00:08:06.646943 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385619 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 00:08:06.646943 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385621 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 00:08:06.646943 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385624 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 00:08:06.646943 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385627 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 00:08:06.646943 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385629 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 00:08:06.646943 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385632 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 00:08:06.646943 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385634 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 00:08:06.646943 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385637 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 00:08:06.646943 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385639 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 00:08:06.646943 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385643 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 00:08:06.646943 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385645 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 00:08:06.646943 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385648 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 00:08:06.647792 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385650 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 00:08:06.647792 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385654 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 00:08:06.647792 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385657 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 00:08:06.647792 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385660 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 00:08:06.647792 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385664 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 00:08:06.647792 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385667 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 00:08:06.647792 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385670 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 00:08:06.647792 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385672 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 00:08:06.647792 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385675 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 00:08:06.647792 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385678 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 00:08:06.647792 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385681 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 00:08:06.647792 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385685 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 00:08:06.647792 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385687 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 00:08:06.647792 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385690 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 00:08:06.647792 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385693 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 00:08:06.647792 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385712 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 00:08:06.647792 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385716 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 00:08:06.647792 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385719 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 00:08:06.647792 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.385722 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 00:08:06.648639 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.385728 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 23 00:08:06.648639 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.393551 2562 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 23 00:08:06.648639 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.393574 2562 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 23 00:08:06.648639 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.394409 2562 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 23 00:08:06.648639 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.394423 2562 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 23 00:08:06.648639 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394508 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 00:08:06.648639 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394514 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 00:08:06.648639 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394518 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 00:08:06.648639 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394521 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 00:08:06.648639 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394524 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 00:08:06.648639 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394527 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 00:08:06.648639 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394529 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 00:08:06.648639 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394533 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 00:08:06.648639 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394536 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 00:08:06.648639 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394538 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 00:08:06.649315 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394541 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 00:08:06.649315 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394544 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 00:08:06.649315 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394547 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 00:08:06.649315 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394550 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 00:08:06.649315 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394552 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 00:08:06.649315 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394555 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 00:08:06.649315 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394557 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 00:08:06.649315 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394560 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 23 00:08:06.649315 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394563 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 00:08:06.649315 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394565 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 00:08:06.649315 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394568 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 00:08:06.649315 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394571 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 00:08:06.649315 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394575 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 00:08:06.649315 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394579 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 00:08:06.649315 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394582 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 00:08:06.649315 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394585 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 00:08:06.649315 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394587 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 00:08:06.649315 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394590 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 00:08:06.649315 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394592 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 00:08:06.649960 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394607 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 00:08:06.649960 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394610 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 00:08:06.649960 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394612 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 00:08:06.649960 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394615 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 00:08:06.649960 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394618 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 00:08:06.649960 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394621 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 00:08:06.649960 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394624 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 00:08:06.649960 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394631 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 00:08:06.649960 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394634 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 00:08:06.649960 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394636 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 00:08:06.649960 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394639 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 00:08:06.649960 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394642 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 00:08:06.649960 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394645 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 00:08:06.649960 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394647 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 00:08:06.649960 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394651 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 00:08:06.649960 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394653 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 00:08:06.649960 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394656 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 00:08:06.649960 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394658 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 00:08:06.649960 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394661 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 00:08:06.649960 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394663 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 00:08:06.650617 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394666 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 00:08:06.650617 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394668 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 00:08:06.650617 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394671 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 00:08:06.650617 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394673 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 00:08:06.650617 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394676 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 00:08:06.650617 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394678 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 00:08:06.650617 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394681 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 00:08:06.650617 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394683 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 00:08:06.650617 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394686 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 00:08:06.650617 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394688 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 00:08:06.650617 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394691 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 00:08:06.650617 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394693 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 00:08:06.650617 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394699 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 00:08:06.650617 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394708 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 00:08:06.650617 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394712 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 00:08:06.650617 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394715 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 00:08:06.650617 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394718 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 00:08:06.650617 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394721 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 00:08:06.650617 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394723 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 00:08:06.651511 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394729 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 00:08:06.651511 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394733 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 00:08:06.651511 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394736 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 00:08:06.651511 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394738 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 00:08:06.651511 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394741 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 00:08:06.651511 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394744 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 00:08:06.651511 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394746 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 00:08:06.651511 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394749 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 00:08:06.651511 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394751 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 00:08:06.651511 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394754 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 00:08:06.651511 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394765 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 00:08:06.651511 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394768 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 00:08:06.651511 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394771 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 00:08:06.651511 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394773 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 00:08:06.651511 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394776 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 00:08:06.651511 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394778 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 00:08:06.651511 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394781 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 00:08:06.651511 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394783 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 00:08:06.651511 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394786 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 00:08:06.652413 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394789 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 00:08:06.652413 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394791 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 00:08:06.652413 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394794 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 00:08:06.652413 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394797 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 00:08:06.652413 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394800 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 00:08:06.652413 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394803 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 00:08:06.652413 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394807 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 00:08:06.652413 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394809 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 00:08:06.652413 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394819 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 00:08:06.652413 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394822 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 00:08:06.652413 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394824 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 00:08:06.652413 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.394827 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 00:08:06.652413 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.394832 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 23 00:08:06.652413 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395033 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 00:08:06.652413 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395040 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 00:08:06.653529 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395044 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 00:08:06.653529 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395048 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 00:08:06.653529 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395051 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 00:08:06.653529 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395054 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 00:08:06.653529 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395057 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 00:08:06.653529 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395060 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 00:08:06.653529 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395063 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 00:08:06.653529 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395065 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 00:08:06.653529 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395068 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 00:08:06.653529 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395071 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 00:08:06.653529 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395073 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 00:08:06.653529 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395076 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 00:08:06.653529 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395078 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 00:08:06.653529 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395081 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 00:08:06.653529 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395083 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 00:08:06.653529 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395086 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 00:08:06.653529 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395089 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 00:08:06.653529 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395091 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 00:08:06.653529 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395094 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 00:08:06.653529 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395096 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 00:08:06.654173 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395099 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 00:08:06.654173 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395101 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 00:08:06.654173 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395104 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 00:08:06.654173 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395106 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 00:08:06.654173 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395111 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 00:08:06.654173 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395114 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 00:08:06.654173 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395124 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 00:08:06.654173 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395127 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 00:08:06.654173 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395131 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 00:08:06.654173 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395134 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 00:08:06.654173 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395139 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 00:08:06.654173 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395142 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 00:08:06.654173 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395145 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 00:08:06.654173 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395148 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 00:08:06.654173 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395151 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 00:08:06.654173 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395154 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 00:08:06.654173 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395156 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 00:08:06.654173 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395159 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 00:08:06.654173 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395162 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 00:08:06.655066 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395165 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 00:08:06.655066 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395167 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 00:08:06.655066 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395170 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 00:08:06.655066 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395172 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 00:08:06.655066 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395175 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 00:08:06.655066 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395177 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 00:08:06.655066 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395180 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 00:08:06.655066 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395182 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 00:08:06.655066 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395185 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 00:08:06.655066 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395188 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 00:08:06.655066 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395190 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 00:08:06.655066 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395193 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 00:08:06.655066 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395195 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 00:08:06.655066 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395198 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 00:08:06.655066 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395200 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 00:08:06.655066 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395203 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 00:08:06.655066 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395205 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 00:08:06.655066 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395209 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 00:08:06.655066 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395212 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 00:08:06.655066 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395215 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 00:08:06.655943 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395223 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 00:08:06.655943 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395226 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 00:08:06.655943 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395229 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 00:08:06.655943 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395233 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 00:08:06.655943 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395236 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 00:08:06.655943 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395239 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 00:08:06.655943 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395241 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 00:08:06.655943 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395244 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 00:08:06.655943 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395246 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 00:08:06.655943 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395249 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 00:08:06.655943 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395251 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 00:08:06.655943 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395255 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 00:08:06.655943 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395259 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 00:08:06.655943 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395262 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 00:08:06.655943 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395265 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 00:08:06.655943 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395267 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 00:08:06.655943 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395270 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 00:08:06.655943 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395273 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 00:08:06.655943 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395275 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 00:08:06.657073 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395278 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 00:08:06.657073 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395281 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 00:08:06.657073 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395283 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 00:08:06.657073 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395286 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 00:08:06.657073 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395288 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 00:08:06.657073 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395291 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 00:08:06.657073 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395293 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 00:08:06.657073 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395296 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 00:08:06.657073 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395298 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 00:08:06.657073 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395301 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 00:08:06.657073 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395306 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 00:08:06.657073 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395308 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 00:08:06.657073 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395311 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 00:08:06.657073 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395313 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 00:08:06.657073 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395322 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 23 00:08:06.657073 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395325 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 00:08:06.657073 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395329 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 00:08:06.657073 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395331 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 00:08:06.657073 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:05.395334 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 00:08:06.658151 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.395339 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 23 00:08:06.658151 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.395532 2562 server.go:962] "Client rotation is on, will bootstrap in background" Jul 23 00:08:06.658151 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.397623 2562 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 23 00:08:06.658151 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.399006 2562 server.go:1019] "Starting client certificate rotation" Jul 23 00:08:06.658151 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.399110 2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 23 00:08:06.658151 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.399157 2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 23 00:08:06.658151 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.402346 2562 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 23 00:08:06.658151 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.434872 2562 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 23 00:08:06.658151 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.447313 2562 log.go:25] "Validated CRI v1 runtime API" Jul 23 00:08:06.658151 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:05.450586 2562 log.go:25] "Validated CRI v1 image API" Jul 23 00:08:06.658151 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.607274 2562 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 23 00:08:06.658151 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.612217 2562 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 87d9dc61-afaa-4060-8df0-24c63dad0a5c:/dev/nvme0n1p4 f303dd85-2b48-4e2e-8b2a-3a1cd2be8eea:/dev/nvme0n1p3] Jul 23 00:08:06.658581 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.612249 2562 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 23 00:08:06.659013 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.617024 2562 manager.go:216] Machine: {Timestamp:2026-07-23 00:08:06.616451239 +0000 UTC m=+0.411958087 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098799 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b2cef8e12656c4026de1b1328beb0 SystemUUID:ec2b2cef-8e12-656c-4026-de1b1328beb0 BootID:3b151b07-f402-4ad1-831b-448b326b3ba5 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:38:ea:14:69:89 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:38:ea:14:69:89 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ea:88:19:bb:7d:71 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 23 00:08:06.659013 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.617144 2562 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 23 00:08:06.659013 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.617237 2562 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 23 00:08:06.659013 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.620153 2562 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 23 00:08:06.659013 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.620215 2562 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-121.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 23 00:08:06.659013 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.621120 2562 topology_manager.go:138] "Creating topology manager with none policy" Jul 23 00:08:06.659013 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.621136 2562 container_manager_linux.go:309] "Creating device plugin manager" Jul 23 00:08:06.659013 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.621149 2562 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 23 00:08:06.659465 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.621859 2562 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 23 00:08:06.659465 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.623068 2562 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 23 00:08:06.659465 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.626128 2562 state_mem.go:36] "Initialized new in-memory state store" Jul 23 00:08:06.659465 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.626278 2562 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 23 00:08:06.659465 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.627060 2562 kubelet.go:486] "Attempting to sync node with API server" Jul 23 00:08:06.659465 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.627072 2562 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 23 00:08:06.659465 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.627089 2562 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 23 00:08:06.659465 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.627103 2562 kubelet.go:398] "Adding apiserver pod source" Jul 23 00:08:06.659465 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.627112 2562 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 23 00:08:06.659465 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.627530 2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 23 00:08:06.659465 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.627548 2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 23 00:08:06.659465 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.630917 2562 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 23 00:08:06.659465 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.632878 2562 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 23 00:08:06.659465 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.632908 2562 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 23 00:08:06.659465 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.634063 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 23 00:08:06.659465 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.634078 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 23 00:08:06.659465 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.634083 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 23 00:08:06.659465 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.634089 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 23 00:08:06.659465 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.634098 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 23 00:08:06.659465 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.634106 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 23 00:08:06.659465 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.634117 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 23 00:08:06.659465 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.634122 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 23 00:08:06.659465 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.634133 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 23 00:08:06.659465 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.634138 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 23 00:08:06.659465 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.634147 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 23 00:08:06.659465 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.634152 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 23 00:08:06.659465 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.634823 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 23 00:08:06.659465 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.634879 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 23 00:08:06.659465 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.638803 2562 server.go:1268] "Started kubelet" Jul 23 00:08:06.659465 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.638947 2562 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 23 00:08:06.659465 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.639078 2562 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 23 00:08:06.659465 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.639015 2562 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 23 00:08:06.659465 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.647307 2562 server.go:310] "Adding debug handlers to kubelet server" Jul 23 00:08:06.659465 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.648981 2562 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 23 00:08:06.659465 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:06.651835 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-121.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 23 00:08:06.659465 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.651884 2562 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-121.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 23 00:08:06.662697 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:06.651780 2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-121.ec2.internal.18c4c292758cf7d8 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-121.ec2.internal,UID:ip-10-0-131-121.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-121.ec2.internal,},FirstTimestamp:2026-07-23 00:08:06.63874556 +0000 UTC m=+0.434252458,LastTimestamp:2026-07-23 00:08:06.63874556 +0000 UTC m=+0.434252458,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-121.ec2.internal,}" Jul 23 00:08:06.662697 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:06.653336 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 23 00:08:06.662697 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.654735 2562 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 23 00:08:06.662697 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.654747 2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 23 00:08:06.662697 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:06.654831 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-121.ec2.internal\" not found" Jul 23 00:08:06.662697 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.655348 2562 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 23 00:08:06.662697 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.655354 2562 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 23 00:08:06.662697 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.655383 2562 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 23 00:08:06.662697 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.655447 2562 reconstruct.go:97] "Volume reconstruction finished" Jul 23 00:08:06.662697 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.655457 2562 reconciler.go:29] "Reconciler: start to sync state" Jul 23 00:08:06.662697 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:06.656752 2562 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-121.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 23 00:08:06.662697 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:06.657101 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 23 00:08:06.722612 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.720038 2562 factory.go:153] Registering CRI-O factory Jul 23 00:08:06.722612 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.720065 2562 factory.go:223] Registration of the crio container factory successfully Jul 23 00:08:06.722612 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.720149 2562 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 23 00:08:06.722612 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.720160 2562 factory.go:55] Registering systemd factory Jul 23 00:08:06.722612 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.720166 2562 factory.go:223] Registration of the systemd container factory successfully Jul 23 00:08:06.722612 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.720187 2562 factory.go:103] Registering Raw factory Jul 23 00:08:06.722612 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.720200 2562 manager.go:1196] Started watching for new ooms in manager Jul 23 00:08:06.722612 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.720727 2562 manager.go:349] Starting recovery of all containers Jul 23 00:08:06.722612 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:06.721133 2562 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 23 00:08:06.732510 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.732208 2562 manager.go:354] Recovery completed Jul 23 00:08:06.789667 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.736819 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 00:08:06.789667 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.738992 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-121.ec2.internal" event="NodeHasSufficientMemory" Jul 23 00:08:06.789667 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.739023 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-121.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 00:08:06.789667 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.739039 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-121.ec2.internal" event="NodeHasSufficientPID" Jul 23 00:08:06.789667 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.739578 2562 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 23 00:08:06.789667 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.739588 2562 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 23 00:08:06.789667 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.739608 2562 state_mem.go:36] "Initialized new in-memory state store" Jul 23 00:08:06.789667 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.739638 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 00:08:06.789667 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.740863 2562 policy_none.go:49] "None policy: Start" Jul 23 00:08:06.789667 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.741284 2562 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 23 00:08:06.789667 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.741297 2562 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 23 00:08:06.789667 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.742535 2562 policy_none.go:47] "Start" Jul 23 00:08:06.789667 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:06.755551 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-121.ec2.internal\" not found" Jul 23 00:08:06.789667 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.776903 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-121.ec2.internal" event="NodeHasSufficientMemory" Jul 23 00:08:06.789667 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.776936 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-121.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 00:08:06.789667 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.776946 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-121.ec2.internal" event="NodeHasSufficientPID" Jul 23 00:08:06.794939 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:06.793848 2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-121.ec2.internal.18c4c2927b86d857 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-121.ec2.internal,UID:ip-10-0-131-121.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-121.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-121.ec2.internal,},FirstTimestamp:2026-07-23 00:08:06.739007575 +0000 UTC m=+0.534514424,LastTimestamp:2026-07-23 00:08:06.739007575 +0000 UTC m=+0.534514424,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-121.ec2.internal,}" Jul 23 00:08:06.874807 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.815296 2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 23 00:08:06.874807 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:06.815282 2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-121.ec2.internal.18c4c2927b8731dd default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-121.ec2.internal,UID:ip-10-0-131-121.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-121.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-121.ec2.internal,},FirstTimestamp:2026-07-23 00:08:06.739030493 +0000 UTC m=+0.534537340,LastTimestamp:2026-07-23 00:08:06.739030493 +0000 UTC m=+0.534537340,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-121.ec2.internal,}" Jul 23 00:08:06.874807 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.815382 2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ssbn2" Jul 23 00:08:06.874807 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:06.817310 2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-121.ec2.internal.18c4c2927b87628e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-121.ec2.internal,UID:ip-10-0-131-121.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-131-121.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-131-121.ec2.internal,},FirstTimestamp:2026-07-23 00:08:06.739042958 +0000 UTC m=+0.534549807,LastTimestamp:2026-07-23 00:08:06.739042958 +0000 UTC m=+0.534549807,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-121.ec2.internal,}" Jul 23 00:08:06.874807 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:06.856147 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-121.ec2.internal\" not found" Jul 23 00:08:06.874807 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.864556 2562 manager.go:337] "Starting Device Plugin manager" Jul 23 00:08:06.874807 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:06.864794 2562 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 23 00:08:06.874807 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.864809 2562 server.go:87] "Starting device plugin registration server" Jul 23 00:08:06.874807 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.865113 2562 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 23 00:08:06.874807 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.865126 2562 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 23 00:08:06.874807 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.865368 2562 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 23 00:08:06.874807 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.865464 2562 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 23 00:08:06.874807 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.865473 2562 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 23 00:08:06.874807 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:06.866019 2562 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 23 00:08:06.874807 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:06.866081 2562 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-121.ec2.internal\" not found" Jul 23 00:08:06.879451 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:06.879354 2562 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-131-121.ec2.internal\" not found" node="ip-10-0-131-121.ec2.internal" Jul 23 00:08:06.974353 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.966252 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 00:08:06.974353 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.967339 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-121.ec2.internal" event="NodeHasSufficientMemory" Jul 23 00:08:06.974353 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.967376 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-121.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 00:08:06.974353 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.967393 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-121.ec2.internal" event="NodeHasSufficientPID" Jul 23 00:08:06.974353 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.967424 2562 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-121.ec2.internal" Jul 23 00:08:06.974353 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.973003 2562 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-121.ec2.internal" Jul 23 00:08:06.974353 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:06.973029 2562 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-121.ec2.internal\": node \"ip-10-0-131-121.ec2.internal\" not found" Jul 23 00:08:07.004093 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:06.995090 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-121.ec2.internal\" not found" Jul 23 00:08:07.004093 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:06.999061 2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 23 00:08:07.004093 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:07.000340 2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 23 00:08:07.004093 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:07.000377 2562 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 23 00:08:07.004093 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:07.000400 2562 kubelet.go:2448] "Starting kubelet main sync loop" Jul 23 00:08:07.004093 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:07.000437 2562 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 23 00:08:07.005422 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:07.004813 2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 00:08:07.107310 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:07.095458 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-121.ec2.internal\" not found" Jul 23 00:08:07.107310 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:07.100545 2562 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-121.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-121.ec2.internal"] Jul 23 00:08:07.107310 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:07.100633 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 00:08:07.107310 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:07.102229 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-121.ec2.internal" event="NodeHasSufficientMemory" Jul 23 00:08:07.107310 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:07.102259 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-121.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 00:08:07.107310 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:07.102271 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-121.ec2.internal" event="NodeHasSufficientPID" Jul 23 00:08:07.198579 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:07.113859 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 00:08:07.198579 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:07.113922 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-121.ec2.internal" Jul 23 00:08:07.198579 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:07.113954 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 00:08:07.198579 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:07.114784 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-121.ec2.internal" event="NodeHasSufficientMemory" Jul 23 00:08:07.198579 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:07.114808 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-121.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 00:08:07.198579 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:07.114785 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-121.ec2.internal" event="NodeHasSufficientMemory" Jul 23 00:08:07.198579 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:07.114839 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-121.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 00:08:07.198579 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:07.114849 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-121.ec2.internal" event="NodeHasSufficientPID" Jul 23 00:08:07.198579 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:07.114817 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-121.ec2.internal" event="NodeHasSufficientPID" Jul 23 00:08:07.198579 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:07.116010 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-121.ec2.internal" Jul 23 00:08:07.198579 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:07.116037 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 00:08:07.198579 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:07.116929 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-121.ec2.internal" event="NodeHasSufficientMemory" Jul 23 00:08:07.198579 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:07.116955 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-121.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 00:08:07.198579 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:07.116969 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-121.ec2.internal" event="NodeHasSufficientPID" Jul 23 00:08:07.198579 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:07.165128 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2dc6c03e0c457691dab513c68ebb04f1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-121.ec2.internal\" (UID: \"2dc6c03e0c457691dab513c68ebb04f1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-121.ec2.internal" Jul 23 00:08:07.198579 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:07.165161 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2dc6c03e0c457691dab513c68ebb04f1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-121.ec2.internal\" (UID: \"2dc6c03e0c457691dab513c68ebb04f1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-121.ec2.internal" Jul 23 00:08:07.198579 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:07.165178 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/19e11d045e1035a525da0259bed898ce-config\") pod \"kube-apiserver-proxy-ip-10-0-131-121.ec2.internal\" (UID: \"19e11d045e1035a525da0259bed898ce\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-121.ec2.internal" Jul 23 00:08:07.198579 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:07.173692 2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-121.ec2.internal\" not found" node="ip-10-0-131-121.ec2.internal" Jul 23 00:08:07.198579 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:07.178554 2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-121.ec2.internal\" not found" node="ip-10-0-131-121.ec2.internal" Jul 23 00:08:07.198579 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:07.196341 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-121.ec2.internal\" not found" Jul 23 00:08:07.272349 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:07.265565 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2dc6c03e0c457691dab513c68ebb04f1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-121.ec2.internal\" (UID: \"2dc6c03e0c457691dab513c68ebb04f1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-121.ec2.internal" Jul 23 00:08:07.272349 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:07.265598 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2dc6c03e0c457691dab513c68ebb04f1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-121.ec2.internal\" (UID: \"2dc6c03e0c457691dab513c68ebb04f1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-121.ec2.internal" Jul 23 00:08:07.272349 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:07.265621 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/19e11d045e1035a525da0259bed898ce-config\") pod \"kube-apiserver-proxy-ip-10-0-131-121.ec2.internal\" (UID: \"19e11d045e1035a525da0259bed898ce\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-121.ec2.internal" Jul 23 00:08:07.272349 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:07.265667 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2dc6c03e0c457691dab513c68ebb04f1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-121.ec2.internal\" (UID: \"2dc6c03e0c457691dab513c68ebb04f1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-121.ec2.internal" Jul 23 00:08:07.272349 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:07.265681 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2dc6c03e0c457691dab513c68ebb04f1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-121.ec2.internal\" (UID: \"2dc6c03e0c457691dab513c68ebb04f1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-121.ec2.internal" Jul 23 00:08:07.272349 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:07.265728 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/19e11d045e1035a525da0259bed898ce-config\") pod \"kube-apiserver-proxy-ip-10-0-131-121.ec2.internal\" (UID: \"19e11d045e1035a525da0259bed898ce\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-121.ec2.internal" Jul 23 00:08:07.371468 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:07.297208 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-121.ec2.internal\" not found" Jul 23 00:08:07.474031 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:07.397717 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-121.ec2.internal\" not found" Jul 23 00:08:07.547050 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:07.477253 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-121.ec2.internal" Jul 23 00:08:07.547050 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:07.481889 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-121.ec2.internal" Jul 23 00:08:07.547050 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:07.498750 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-121.ec2.internal\" not found" Jul 23 00:08:07.550918 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:07.550894 2562 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 23 00:08:07.551276 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:07.551093 2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 23 00:08:07.599491 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:07.599457 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-121.ec2.internal\" not found" Jul 23 00:08:07.654650 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:07.654571 2562 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 00:08:07.673509 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:07.673471 2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 00:08:07.700619 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:07.700586 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-121.ec2.internal\" not found" Jul 23 00:08:07.797010 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:07.796978 2562 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 00:08:07.821560 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:07.821498 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-22 00:03:06 +0000 UTC" deadline="2028-04-23 11:12:26.404083096 +0000 UTC" Jul 23 00:08:07.821560 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:07.821550 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15371h4m18.582536767s" Jul 23 00:08:07.855733 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:07.855698 2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-121.ec2.internal" Jul 23 00:08:07.867024 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:07.866998 2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 23 00:08:07.867993 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:07.867978 2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-121.ec2.internal" Jul 23 00:08:07.877345 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:07.877318 2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 23 00:08:08.056799 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:08.056746 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19e11d045e1035a525da0259bed898ce.slice/crio-237d330bb1f91846b8ba7dc06f544b1d29c48d77c5d49d51b2e2305df3c5d78e WatchSource:0}: Error finding container 237d330bb1f91846b8ba7dc06f544b1d29c48d77c5d49d51b2e2305df3c5d78e: Status 404 returned error can't find the container with id 237d330bb1f91846b8ba7dc06f544b1d29c48d77c5d49d51b2e2305df3c5d78e Jul 23 00:08:08.057131 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:08.057108 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dc6c03e0c457691dab513c68ebb04f1.slice/crio-d457c572e3527be958bee3d3cea9b19f1566feca3f504369ad1ffa3f74826d85 WatchSource:0}: Error finding container d457c572e3527be958bee3d3cea9b19f1566feca3f504369ad1ffa3f74826d85: Status 404 returned error can't find the container with id d457c572e3527be958bee3d3cea9b19f1566feca3f504369ad1ffa3f74826d85 Jul 23 00:08:08.062997 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.062978 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 00:08:08.628068 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.627936 2562 apiserver.go:52] "Watching apiserver" Jul 23 00:08:08.641256 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.641215 2562 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 23 00:08:08.643110 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.643070 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-cgvgd","kube-system/global-pull-secret-syncer-jlsd6","openshift-cluster-node-tuning-operator/tuned-sr9sh","openshift-multus/multus-additional-cni-plugins-8ghxf","openshift-multus/network-metrics-daemon-kfgzz","openshift-network-operator/iptables-alerter-626nn","openshift-ovn-kubernetes/ovnkube-node-bjc8c","kube-system/konnectivity-agent-fhbjd","kube-system/kube-apiserver-proxy-ip-10-0-131-121.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvpdz","openshift-dns/node-resolver-9stwt","openshift-image-registry/node-ca-jctrc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-121.ec2.internal","openshift-multus/multus-vgrs2"] Jul 23 00:08:08.645510 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.645467 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-sr9sh" Jul 23 00:08:08.647336 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.647313 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8ghxf" Jul 23 00:08:08.648822 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.648362 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 23 00:08:08.648822 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.648704 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zmzbv\"" Jul 23 00:08:08.648986 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.648948 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 23 00:08:08.649213 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.649191 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" Jul 23 00:08:08.650673 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.650628 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jlsd6" Jul 23 00:08:08.650793 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:08.650706 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jlsd6" podUID="5f272157-a81d-439c-86ed-511e11146698" Jul 23 00:08:08.651525 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.650934 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 23 00:08:08.651525 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.651013 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bccq7\"" Jul 23 00:08:08.651525 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.651221 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 23 00:08:08.651525 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.651346 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 23 00:08:08.651525 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.651398 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 23 00:08:08.651525 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.651428 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 23 00:08:08.652223 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.652198 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 23 00:08:08.652711 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.652422 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cbgfl\"" Jul 23 00:08:08.652711 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.652614 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 23 00:08:08.652879 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.652848 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 23 00:08:08.653132 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.653017 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfgzz" Jul 23 00:08:08.653132 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:08.653091 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfgzz" podUID="5a9bdee1-a493-4223-a7b6-3ae1f5024eb6" Jul 23 00:08:08.653264 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.653138 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 23 00:08:08.654221 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.653939 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 23 00:08:08.654221 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.654017 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 23 00:08:08.654379 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.654227 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-626nn" Jul 23 00:08:08.656799 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.655044 2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 23 00:08:08.656799 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.656516 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cgvgd" Jul 23 00:08:08.656799 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:08.656579 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cgvgd" podUID="72c0b00d-f59f-4186-b84e-470a1adac941" Jul 23 00:08:08.658592 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.657403 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fhbjd" Jul 23 00:08:08.658592 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.658163 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 23 00:08:08.658592 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.658262 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 23 00:08:08.658592 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.658381 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-kw644\"" Jul 23 00:08:08.658592 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.658449 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 23 00:08:08.659104 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.659082 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvpdz" Jul 23 00:08:08.660610 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.660429 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-x5smg\"" Jul 23 00:08:08.660952 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.660934 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 23 00:08:08.662061 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.661346 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 23 00:08:08.662061 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.661693 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9stwt" Jul 23 00:08:08.662227 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.662160 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jctrc" Jul 23 00:08:08.662227 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.662178 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 23 00:08:08.662227 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.662221 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 23 00:08:08.662461 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.662440 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 23 00:08:08.662630 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.662607 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-h9gf4\"" Jul 23 00:08:08.663730 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.663710 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vgrs2" Jul 23 00:08:08.665915 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.665895 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 23 00:08:08.666017 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.665958 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 23 00:08:08.666078 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.666055 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ltqsf\"" Jul 23 00:08:08.666166 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.666149 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 23 00:08:08.667296 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.666347 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mpw27\"" Jul 23 00:08:08.667296 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.666391 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 23 00:08:08.667296 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.666667 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-txrt7\"" Jul 23 00:08:08.667296 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.666793 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 23 00:08:08.667296 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.667032 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 23 00:08:08.670588 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.670471 2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 23 00:08:08.671403 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.671369 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/af5c6a95-1825-4855-9efb-906c85d29aa5-var-lib-kubelet\") pod \"tuned-sr9sh\" (UID: \"af5c6a95-1825-4855-9efb-906c85d29aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9sh" Jul 23 00:08:08.671499 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.671408 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bts77\" (UniqueName: \"kubernetes.io/projected/af5c6a95-1825-4855-9efb-906c85d29aa5-kube-api-access-bts77\") pod \"tuned-sr9sh\" (UID: \"af5c6a95-1825-4855-9efb-906c85d29aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9sh" Jul 23 00:08:08.671499 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.671437 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/224cde93-d872-474d-b1c4-c9f92fc1f622-host-slash\") pod \"ovnkube-node-bjc8c\" (UID: \"224cde93-d872-474d-b1c4-c9f92fc1f622\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" Jul 23 00:08:08.671499 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.671469 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/224cde93-d872-474d-b1c4-c9f92fc1f622-host-run-netns\") pod \"ovnkube-node-bjc8c\" (UID: \"224cde93-d872-474d-b1c4-c9f92fc1f622\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" Jul 23 00:08:08.671499 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.671495 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/224cde93-d872-474d-b1c4-c9f92fc1f622-etc-openvswitch\") pod \"ovnkube-node-bjc8c\" (UID: \"224cde93-d872-474d-b1c4-c9f92fc1f622\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" Jul 23 00:08:08.671707 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.671519 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/224cde93-d872-474d-b1c4-c9f92fc1f622-ovnkube-script-lib\") pod \"ovnkube-node-bjc8c\" (UID: \"224cde93-d872-474d-b1c4-c9f92fc1f622\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" Jul 23 00:08:08.671707 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.671552 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/af5c6a95-1825-4855-9efb-906c85d29aa5-etc-sysctl-conf\") pod \"tuned-sr9sh\" (UID: \"af5c6a95-1825-4855-9efb-906c85d29aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9sh" Jul 23 00:08:08.671707 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.671575 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/af5c6a95-1825-4855-9efb-906c85d29aa5-host\") pod \"tuned-sr9sh\" (UID: \"af5c6a95-1825-4855-9efb-906c85d29aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9sh" Jul 23 00:08:08.671707 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.671598 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af5c6a95-1825-4855-9efb-906c85d29aa5-tmp\") pod \"tuned-sr9sh\" (UID: \"af5c6a95-1825-4855-9efb-906c85d29aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9sh" Jul 23 00:08:08.671707 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.671622 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/224cde93-d872-474d-b1c4-c9f92fc1f622-systemd-units\") pod \"ovnkube-node-bjc8c\" (UID: \"224cde93-d872-474d-b1c4-c9f92fc1f622\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" Jul 23 00:08:08.671707 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.671644 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/224cde93-d872-474d-b1c4-c9f92fc1f622-host-cni-bin\") pod \"ovnkube-node-bjc8c\" (UID: \"224cde93-d872-474d-b1c4-c9f92fc1f622\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" Jul 23 00:08:08.671707 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.671667 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/224cde93-d872-474d-b1c4-c9f92fc1f622-ovn-node-metrics-cert\") pod \"ovnkube-node-bjc8c\" (UID: \"224cde93-d872-474d-b1c4-c9f92fc1f622\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" Jul 23 00:08:08.671707 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.671692 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwcm6\" (UniqueName: \"kubernetes.io/projected/5a9bdee1-a493-4223-a7b6-3ae1f5024eb6-kube-api-access-kwcm6\") pod \"network-metrics-daemon-kfgzz\" (UID: \"5a9bdee1-a493-4223-a7b6-3ae1f5024eb6\") " pod="openshift-multus/network-metrics-daemon-kfgzz" Jul 23 00:08:08.672114 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.671748 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f9853266-a585-469c-885e-79cc72f88d18-iptables-alerter-script\") pod \"iptables-alerter-626nn\" (UID: \"f9853266-a585-469c-885e-79cc72f88d18\") " pod="openshift-network-operator/iptables-alerter-626nn" Jul 23 00:08:08.672114 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.671852 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/af5c6a95-1825-4855-9efb-906c85d29aa5-run\") pod \"tuned-sr9sh\" (UID: \"af5c6a95-1825-4855-9efb-906c85d29aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9sh" Jul 23 00:08:08.672114 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.671890 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/224cde93-d872-474d-b1c4-c9f92fc1f622-log-socket\") pod \"ovnkube-node-bjc8c\" (UID: \"224cde93-d872-474d-b1c4-c9f92fc1f622\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" Jul 23 00:08:08.672114 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.671939 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzxdr\" (UniqueName: \"kubernetes.io/projected/f9853266-a585-469c-885e-79cc72f88d18-kube-api-access-lzxdr\") pod \"iptables-alerter-626nn\" (UID: \"f9853266-a585-469c-885e-79cc72f88d18\") " pod="openshift-network-operator/iptables-alerter-626nn" Jul 23 00:08:08.672114 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.671979 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/216d84cd-f540-4bc0-9bd1-a8cc9e37367f-os-release\") pod \"multus-additional-cni-plugins-8ghxf\" (UID: \"216d84cd-f540-4bc0-9bd1-a8cc9e37367f\") " pod="openshift-multus/multus-additional-cni-plugins-8ghxf" Jul 23 00:08:08.672114 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.672007 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/216d84cd-f540-4bc0-9bd1-a8cc9e37367f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8ghxf\" (UID: \"216d84cd-f540-4bc0-9bd1-a8cc9e37367f\") " pod="openshift-multus/multus-additional-cni-plugins-8ghxf" Jul 23 00:08:08.672114 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.672031 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f272157-a81d-439c-86ed-511e11146698-original-pull-secret\") pod \"global-pull-secret-syncer-jlsd6\" (UID: \"5f272157-a81d-439c-86ed-511e11146698\") " pod="kube-system/global-pull-secret-syncer-jlsd6" Jul 23 00:08:08.672114 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.672080 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af5c6a95-1825-4855-9efb-906c85d29aa5-etc-kubernetes\") pod \"tuned-sr9sh\" (UID: \"af5c6a95-1825-4855-9efb-906c85d29aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9sh" Jul 23 00:08:08.672114 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.672104 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/af5c6a95-1825-4855-9efb-906c85d29aa5-sys\") pod \"tuned-sr9sh\" (UID: \"af5c6a95-1825-4855-9efb-906c85d29aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9sh" Jul 23 00:08:08.672114 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.672119 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/af5c6a95-1825-4855-9efb-906c85d29aa5-etc-tuned\") pod \"tuned-sr9sh\" (UID: \"af5c6a95-1825-4855-9efb-906c85d29aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9sh" Jul 23 00:08:08.672618 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.672159 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/224cde93-d872-474d-b1c4-c9f92fc1f622-run-openvswitch\") pod \"ovnkube-node-bjc8c\" (UID: \"224cde93-d872-474d-b1c4-c9f92fc1f622\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" Jul 23 00:08:08.672618 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.672180 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/224cde93-d872-474d-b1c4-c9f92fc1f622-run-ovn\") pod \"ovnkube-node-bjc8c\" (UID: \"224cde93-d872-474d-b1c4-c9f92fc1f622\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" Jul 23 00:08:08.672618 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.672209 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/224cde93-d872-474d-b1c4-c9f92fc1f622-node-log\") pod \"ovnkube-node-bjc8c\" (UID: \"224cde93-d872-474d-b1c4-c9f92fc1f622\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" Jul 23 00:08:08.672618 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.672231 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/224cde93-d872-474d-b1c4-c9f92fc1f622-ovnkube-config\") pod \"ovnkube-node-bjc8c\" (UID: \"224cde93-d872-474d-b1c4-c9f92fc1f622\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" Jul 23 00:08:08.672618 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.672271 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/224cde93-d872-474d-b1c4-c9f92fc1f622-env-overrides\") pod \"ovnkube-node-bjc8c\" (UID: \"224cde93-d872-474d-b1c4-c9f92fc1f622\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" Jul 23 00:08:08.672618 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.672294 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/af5c6a95-1825-4855-9efb-906c85d29aa5-lib-modules\") pod \"tuned-sr9sh\" (UID: \"af5c6a95-1825-4855-9efb-906c85d29aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9sh" Jul 23 00:08:08.672618 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.672315 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/224cde93-d872-474d-b1c4-c9f92fc1f622-var-lib-openvswitch\") pod \"ovnkube-node-bjc8c\" (UID: \"224cde93-d872-474d-b1c4-c9f92fc1f622\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" Jul 23 00:08:08.672618 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.672339 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/224cde93-d872-474d-b1c4-c9f92fc1f622-host-cni-netd\") pod \"ovnkube-node-bjc8c\" (UID: \"224cde93-d872-474d-b1c4-c9f92fc1f622\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" Jul 23 00:08:08.672618 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.672361 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5x4x\" (UniqueName: \"kubernetes.io/projected/224cde93-d872-474d-b1c4-c9f92fc1f622-kube-api-access-g5x4x\") pod \"ovnkube-node-bjc8c\" (UID: \"224cde93-d872-474d-b1c4-c9f92fc1f622\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" Jul 23 00:08:08.672618 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.672386 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a9bdee1-a493-4223-a7b6-3ae1f5024eb6-metrics-certs\") pod \"network-metrics-daemon-kfgzz\" (UID: \"5a9bdee1-a493-4223-a7b6-3ae1f5024eb6\") " pod="openshift-multus/network-metrics-daemon-kfgzz" Jul 23 00:08:08.672618 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.672408 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/216d84cd-f540-4bc0-9bd1-a8cc9e37367f-cnibin\") pod \"multus-additional-cni-plugins-8ghxf\" (UID: \"216d84cd-f540-4bc0-9bd1-a8cc9e37367f\") " pod="openshift-multus/multus-additional-cni-plugins-8ghxf" Jul 23 00:08:08.672618 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.672446 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpdz6\" (UniqueName: \"kubernetes.io/projected/216d84cd-f540-4bc0-9bd1-a8cc9e37367f-kube-api-access-cpdz6\") pod \"multus-additional-cni-plugins-8ghxf\" (UID: \"216d84cd-f540-4bc0-9bd1-a8cc9e37367f\") " pod="openshift-multus/multus-additional-cni-plugins-8ghxf" Jul 23 00:08:08.672618 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.672513 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/af5c6a95-1825-4855-9efb-906c85d29aa5-etc-modprobe-d\") pod \"tuned-sr9sh\" (UID: \"af5c6a95-1825-4855-9efb-906c85d29aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9sh" Jul 23 00:08:08.672618 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.672536 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/af5c6a95-1825-4855-9efb-906c85d29aa5-etc-sysctl-d\") pod \"tuned-sr9sh\" (UID: \"af5c6a95-1825-4855-9efb-906c85d29aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9sh" Jul 23 00:08:08.673276 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.672560 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/216d84cd-f540-4bc0-9bd1-a8cc9e37367f-cni-binary-copy\") pod \"multus-additional-cni-plugins-8ghxf\" (UID: \"216d84cd-f540-4bc0-9bd1-a8cc9e37367f\") " pod="openshift-multus/multus-additional-cni-plugins-8ghxf" Jul 23 00:08:08.673276 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.672584 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/216d84cd-f540-4bc0-9bd1-a8cc9e37367f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8ghxf\" (UID: \"216d84cd-f540-4bc0-9bd1-a8cc9e37367f\") " pod="openshift-multus/multus-additional-cni-plugins-8ghxf" Jul 23 00:08:08.673276 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.672614 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/af5c6a95-1825-4855-9efb-906c85d29aa5-etc-sysconfig\") pod \"tuned-sr9sh\" (UID: \"af5c6a95-1825-4855-9efb-906c85d29aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9sh" Jul 23 00:08:08.673276 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.672643 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/224cde93-d872-474d-b1c4-c9f92fc1f622-host-kubelet\") pod \"ovnkube-node-bjc8c\" (UID: \"224cde93-d872-474d-b1c4-c9f92fc1f622\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" Jul 23 00:08:08.673276 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.672665 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/224cde93-d872-474d-b1c4-c9f92fc1f622-run-systemd\") pod \"ovnkube-node-bjc8c\" (UID: \"224cde93-d872-474d-b1c4-c9f92fc1f622\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" Jul 23 00:08:08.673276 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.672690 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/224cde93-d872-474d-b1c4-c9f92fc1f622-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bjc8c\" (UID: \"224cde93-d872-474d-b1c4-c9f92fc1f622\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" Jul 23 00:08:08.673276 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.672725 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f9853266-a585-469c-885e-79cc72f88d18-host-slash\") pod \"iptables-alerter-626nn\" (UID: \"f9853266-a585-469c-885e-79cc72f88d18\") " pod="openshift-network-operator/iptables-alerter-626nn" Jul 23 00:08:08.673276 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.672748 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2thdl\" (UniqueName: \"kubernetes.io/projected/72c0b00d-f59f-4186-b84e-470a1adac941-kube-api-access-2thdl\") pod \"network-check-target-cgvgd\" (UID: \"72c0b00d-f59f-4186-b84e-470a1adac941\") " pod="openshift-network-diagnostics/network-check-target-cgvgd" Jul 23 00:08:08.673276 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.672800 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/216d84cd-f540-4bc0-9bd1-a8cc9e37367f-system-cni-dir\") pod \"multus-additional-cni-plugins-8ghxf\" (UID: \"216d84cd-f540-4bc0-9bd1-a8cc9e37367f\") " pod="openshift-multus/multus-additional-cni-plugins-8ghxf" Jul 23 00:08:08.673276 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.672828 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5f272157-a81d-439c-86ed-511e11146698-kubelet-config\") pod \"global-pull-secret-syncer-jlsd6\" (UID: \"5f272157-a81d-439c-86ed-511e11146698\") " pod="kube-system/global-pull-secret-syncer-jlsd6" Jul 23 00:08:08.673276 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.672853 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/af5c6a95-1825-4855-9efb-906c85d29aa5-etc-systemd\") pod \"tuned-sr9sh\" (UID: \"af5c6a95-1825-4855-9efb-906c85d29aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9sh" Jul 23 00:08:08.673276 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.672915 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/224cde93-d872-474d-b1c4-c9f92fc1f622-host-run-ovn-kubernetes\") pod \"ovnkube-node-bjc8c\" (UID: \"224cde93-d872-474d-b1c4-c9f92fc1f622\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" Jul 23 00:08:08.673276 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.672939 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/216d84cd-f540-4bc0-9bd1-a8cc9e37367f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8ghxf\" (UID: \"216d84cd-f540-4bc0-9bd1-a8cc9e37367f\") " pod="openshift-multus/multus-additional-cni-plugins-8ghxf" Jul 23 00:08:08.673924 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.672960 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5f272157-a81d-439c-86ed-511e11146698-dbus\") pod \"global-pull-secret-syncer-jlsd6\" (UID: \"5f272157-a81d-439c-86ed-511e11146698\") " pod="kube-system/global-pull-secret-syncer-jlsd6" Jul 23 00:08:08.691028 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.690983 2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-gkv54" Jul 23 00:08:08.698189 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.698155 2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-gkv54" Jul 23 00:08:08.756314 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.756283 2562 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 23 00:08:08.773215 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.773179 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d457c50-6bca-4609-8897-cef340b087e8-tmp-dir\") pod \"node-resolver-9stwt\" (UID: \"7d457c50-6bca-4609-8897-cef340b087e8\") " pod="openshift-dns/node-resolver-9stwt" Jul 23 00:08:08.773215 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.773222 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3749b807-a46e-43b6-880a-ac9bec9200f6-device-dir\") pod \"aws-ebs-csi-driver-node-mvpdz\" (UID: \"3749b807-a46e-43b6-880a-ac9bec9200f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvpdz" Jul 23 00:08:08.773451 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.773246 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9b166998-6f59-4909-b0aa-4c015ce61565-os-release\") pod \"multus-vgrs2\" (UID: \"9b166998-6f59-4909-b0aa-4c015ce61565\") " pod="openshift-multus/multus-vgrs2" Jul 23 00:08:08.773451 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.773271 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9b166998-6f59-4909-b0aa-4c015ce61565-host-var-lib-cni-multus\") pod \"multus-vgrs2\" (UID: \"9b166998-6f59-4909-b0aa-4c015ce61565\") " pod="openshift-multus/multus-vgrs2" Jul 23 00:08:08.773451 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.773324 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/af5c6a95-1825-4855-9efb-906c85d29aa5-etc-systemd\") pod \"tuned-sr9sh\" (UID: \"af5c6a95-1825-4855-9efb-906c85d29aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9sh" Jul 23 00:08:08.773451 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.773349 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/224cde93-d872-474d-b1c4-c9f92fc1f622-host-run-ovn-kubernetes\") pod \"ovnkube-node-bjc8c\" (UID: \"224cde93-d872-474d-b1c4-c9f92fc1f622\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" Jul 23 00:08:08.773451 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.773372 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7d457c50-6bca-4609-8897-cef340b087e8-hosts-file\") pod \"node-resolver-9stwt\" (UID: \"7d457c50-6bca-4609-8897-cef340b087e8\") " pod="openshift-dns/node-resolver-9stwt" Jul 23 00:08:08.773451 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.773399 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/224cde93-d872-474d-b1c4-c9f92fc1f622-host-run-netns\") pod \"ovnkube-node-bjc8c\" (UID: \"224cde93-d872-474d-b1c4-c9f92fc1f622\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" Jul 23 00:08:08.773451 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.773423 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/224cde93-d872-474d-b1c4-c9f92fc1f622-ovnkube-script-lib\") pod \"ovnkube-node-bjc8c\" (UID: \"224cde93-d872-474d-b1c4-c9f92fc1f622\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" Jul 23 00:08:08.773860 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.773452 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/af5c6a95-1825-4855-9efb-906c85d29aa5-etc-systemd\") pod \"tuned-sr9sh\" (UID: \"af5c6a95-1825-4855-9efb-906c85d29aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9sh" Jul 23 00:08:08.773860 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.773509 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/19f5db96-5df5-4b0c-8cc1-89d3f2719ed2-konnectivity-ca\") pod \"konnectivity-agent-fhbjd\" (UID: \"19f5db96-5df5-4b0c-8cc1-89d3f2719ed2\") " pod="kube-system/konnectivity-agent-fhbjd" Jul 23 00:08:08.773860 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.773579 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/224cde93-d872-474d-b1c4-c9f92fc1f622-host-run-netns\") pod \"ovnkube-node-bjc8c\" (UID: \"224cde93-d872-474d-b1c4-c9f92fc1f622\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" Jul 23 00:08:08.773860 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.773584 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/224cde93-d872-474d-b1c4-c9f92fc1f622-host-run-ovn-kubernetes\") pod \"ovnkube-node-bjc8c\" (UID: \"224cde93-d872-474d-b1c4-c9f92fc1f622\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" Jul 23 00:08:08.773860 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.773614 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3749b807-a46e-43b6-880a-ac9bec9200f6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mvpdz\" (UID: \"3749b807-a46e-43b6-880a-ac9bec9200f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvpdz" Jul 23 00:08:08.773860 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.773632 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3749b807-a46e-43b6-880a-ac9bec9200f6-socket-dir\") pod \"aws-ebs-csi-driver-node-mvpdz\" (UID: \"3749b807-a46e-43b6-880a-ac9bec9200f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvpdz" Jul 23 00:08:08.773860 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.773649 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b166998-6f59-4909-b0aa-4c015ce61565-host-var-lib-kubelet\") pod \"multus-vgrs2\" (UID: \"9b166998-6f59-4909-b0aa-4c015ce61565\") " pod="openshift-multus/multus-vgrs2" Jul 23 00:08:08.773860 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.773670 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/af5c6a95-1825-4855-9efb-906c85d29aa5-etc-sysctl-conf\") pod \"tuned-sr9sh\" (UID: \"af5c6a95-1825-4855-9efb-906c85d29aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9sh" Jul 23 00:08:08.773860 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.773693 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/216d84cd-f540-4bc0-9bd1-a8cc9e37367f-os-release\") pod \"multus-additional-cni-plugins-8ghxf\" (UID: \"216d84cd-f540-4bc0-9bd1-a8cc9e37367f\") " pod="openshift-multus/multus-additional-cni-plugins-8ghxf" Jul 23 00:08:08.773860 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.773714 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9b166998-6f59-4909-b0aa-4c015ce61565-multus-socket-dir-parent\") pod \"multus-vgrs2\" (UID: \"9b166998-6f59-4909-b0aa-4c015ce61565\") " pod="openshift-multus/multus-vgrs2" Jul 23 00:08:08.773860 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.773782 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz4qf\" (UniqueName: \"kubernetes.io/projected/9b166998-6f59-4909-b0aa-4c015ce61565-kube-api-access-bz4qf\") pod \"multus-vgrs2\" (UID: \"9b166998-6f59-4909-b0aa-4c015ce61565\") " pod="openshift-multus/multus-vgrs2" Jul 23 00:08:08.773860 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.773801 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3749b807-a46e-43b6-880a-ac9bec9200f6-etc-selinux\") pod \"aws-ebs-csi-driver-node-mvpdz\" (UID: \"3749b807-a46e-43b6-880a-ac9bec9200f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvpdz" Jul 23 00:08:08.773860 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.773803 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/216d84cd-f540-4bc0-9bd1-a8cc9e37367f-os-release\") pod \"multus-additional-cni-plugins-8ghxf\" (UID: \"216d84cd-f540-4bc0-9bd1-a8cc9e37367f\") " pod="openshift-multus/multus-additional-cni-plugins-8ghxf" Jul 23 00:08:08.773860 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.773816 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b166998-6f59-4909-b0aa-4c015ce61565-etc-kubernetes\") pod \"multus-vgrs2\" (UID: \"9b166998-6f59-4909-b0aa-4c015ce61565\") " pod="openshift-multus/multus-vgrs2" Jul 23 00:08:08.774541 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.773848 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af5c6a95-1825-4855-9efb-906c85d29aa5-etc-kubernetes\") pod \"tuned-sr9sh\" (UID: \"af5c6a95-1825-4855-9efb-906c85d29aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9sh" Jul 23 00:08:08.774541 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.773863 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/af5c6a95-1825-4855-9efb-906c85d29aa5-etc-sysctl-conf\") pod \"tuned-sr9sh\" (UID: \"af5c6a95-1825-4855-9efb-906c85d29aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9sh" Jul 23 00:08:08.774541 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.773893 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af5c6a95-1825-4855-9efb-906c85d29aa5-etc-kubernetes\") pod \"tuned-sr9sh\" (UID: \"af5c6a95-1825-4855-9efb-906c85d29aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9sh" Jul 23 00:08:08.774541 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.773909 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/af5c6a95-1825-4855-9efb-906c85d29aa5-sys\") pod \"tuned-sr9sh\" (UID: \"af5c6a95-1825-4855-9efb-906c85d29aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9sh" Jul 23 00:08:08.774541 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.773943 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/af5c6a95-1825-4855-9efb-906c85d29aa5-etc-tuned\") pod \"tuned-sr9sh\" (UID: \"af5c6a95-1825-4855-9efb-906c85d29aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9sh" Jul 23 00:08:08.774541 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.774006 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/224cde93-d872-474d-b1c4-c9f92fc1f622-run-openvswitch\") pod \"ovnkube-node-bjc8c\" (UID: \"224cde93-d872-474d-b1c4-c9f92fc1f622\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" Jul 23 00:08:08.774541 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.774035 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/224cde93-d872-474d-b1c4-c9f92fc1f622-node-log\") pod \"ovnkube-node-bjc8c\" (UID: \"224cde93-d872-474d-b1c4-c9f92fc1f622\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" Jul 23 00:08:08.774541 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.774040 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/af5c6a95-1825-4855-9efb-906c85d29aa5-sys\") pod \"tuned-sr9sh\" (UID: \"af5c6a95-1825-4855-9efb-906c85d29aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9sh" Jul 23 00:08:08.774541 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.774052 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/224cde93-d872-474d-b1c4-c9f92fc1f622-run-openvswitch\") pod \"ovnkube-node-bjc8c\" (UID: \"224cde93-d872-474d-b1c4-c9f92fc1f622\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" Jul 23 00:08:08.774541 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.774069 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5x4x\" (UniqueName: \"kubernetes.io/projected/224cde93-d872-474d-b1c4-c9f92fc1f622-kube-api-access-g5x4x\") pod \"ovnkube-node-bjc8c\" (UID: \"224cde93-d872-474d-b1c4-c9f92fc1f622\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" Jul 23 00:08:08.774541 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.774074 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/224cde93-d872-474d-b1c4-c9f92fc1f622-node-log\") pod \"ovnkube-node-bjc8c\" (UID: \"224cde93-d872-474d-b1c4-c9f92fc1f622\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" Jul 23 00:08:08.774541 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.774097 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b166998-6f59-4909-b0aa-4c015ce61565-system-cni-dir\") pod \"multus-vgrs2\" (UID: \"9b166998-6f59-4909-b0aa-4c015ce61565\") " pod="openshift-multus/multus-vgrs2" Jul 23 00:08:08.774541 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.774122 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/224cde93-d872-474d-b1c4-c9f92fc1f622-var-lib-openvswitch\") pod \"ovnkube-node-bjc8c\" (UID: \"224cde93-d872-474d-b1c4-c9f92fc1f622\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" Jul 23 00:08:08.774541 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.774146 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/224cde93-d872-474d-b1c4-c9f92fc1f622-host-cni-netd\") pod \"ovnkube-node-bjc8c\" (UID: \"224cde93-d872-474d-b1c4-c9f92fc1f622\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" Jul 23 00:08:08.774541 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.774169 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/216d84cd-f540-4bc0-9bd1-a8cc9e37367f-cnibin\") pod \"multus-additional-cni-plugins-8ghxf\" (UID: \"216d84cd-f540-4bc0-9bd1-a8cc9e37367f\") " pod="openshift-multus/multus-additional-cni-plugins-8ghxf" Jul 23 00:08:08.775342 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.774178 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/224cde93-d872-474d-b1c4-c9f92fc1f622-var-lib-openvswitch\") pod \"ovnkube-node-bjc8c\" (UID: \"224cde93-d872-474d-b1c4-c9f92fc1f622\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" Jul 23 00:08:08.775342 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.774192 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b166998-6f59-4909-b0aa-4c015ce61565-multus-cni-dir\") pod \"multus-vgrs2\" (UID: \"9b166998-6f59-4909-b0aa-4c015ce61565\") " pod="openshift-multus/multus-vgrs2" Jul 23 00:08:08.775342 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.774215 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9b166998-6f59-4909-b0aa-4c015ce61565-host-var-lib-cni-bin\") pod \"multus-vgrs2\" (UID: \"9b166998-6f59-4909-b0aa-4c015ce61565\") " pod="openshift-multus/multus-vgrs2" Jul 23 00:08:08.775342 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.774233 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/224cde93-d872-474d-b1c4-c9f92fc1f622-host-cni-netd\") pod \"ovnkube-node-bjc8c\" (UID: \"224cde93-d872-474d-b1c4-c9f92fc1f622\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" Jul 23 00:08:08.775342 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.774238 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9b166998-6f59-4909-b0aa-4c015ce61565-hostroot\") pod \"multus-vgrs2\" (UID: \"9b166998-6f59-4909-b0aa-4c015ce61565\") " pod="openshift-multus/multus-vgrs2" Jul 23 00:08:08.775342 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.774270 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9b166998-6f59-4909-b0aa-4c015ce61565-multus-conf-dir\") pod \"multus-vgrs2\" (UID: \"9b166998-6f59-4909-b0aa-4c015ce61565\") " pod="openshift-multus/multus-vgrs2" Jul 23 00:08:08.775342 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.774256 2562 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 23 00:08:08.775342 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.774290 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/216d84cd-f540-4bc0-9bd1-a8cc9e37367f-cnibin\") pod \"multus-additional-cni-plugins-8ghxf\" (UID: \"216d84cd-f540-4bc0-9bd1-a8cc9e37367f\") " pod="openshift-multus/multus-additional-cni-plugins-8ghxf" Jul 23 00:08:08.775342 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.774298 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/af5c6a95-1825-4855-9efb-906c85d29aa5-etc-modprobe-d\") pod \"tuned-sr9sh\" (UID: \"af5c6a95-1825-4855-9efb-906c85d29aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9sh" Jul 23 00:08:08.775342 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.774322 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/224cde93-d872-474d-b1c4-c9f92fc1f622-ovnkube-script-lib\") pod \"ovnkube-node-bjc8c\" (UID: \"224cde93-d872-474d-b1c4-c9f92fc1f622\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" Jul 23 00:08:08.775342 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.774333 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/216d84cd-f540-4bc0-9bd1-a8cc9e37367f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8ghxf\" (UID: \"216d84cd-f540-4bc0-9bd1-a8cc9e37367f\") " pod="openshift-multus/multus-additional-cni-plugins-8ghxf" Jul 23 00:08:08.775342 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.774364 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3749b807-a46e-43b6-880a-ac9bec9200f6-sys-fs\") pod \"aws-ebs-csi-driver-node-mvpdz\" (UID: \"3749b807-a46e-43b6-880a-ac9bec9200f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvpdz" Jul 23 00:08:08.775342 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.774390 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9b166998-6f59-4909-b0aa-4c015ce61565-host-run-multus-certs\") pod \"multus-vgrs2\" (UID: \"9b166998-6f59-4909-b0aa-4c015ce61565\") " pod="openshift-multus/multus-vgrs2" Jul 23 00:08:08.775342 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.774400 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/af5c6a95-1825-4855-9efb-906c85d29aa5-etc-modprobe-d\") pod \"tuned-sr9sh\" (UID: \"af5c6a95-1825-4855-9efb-906c85d29aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9sh" Jul 23 00:08:08.775342 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.774417 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/224cde93-d872-474d-b1c4-c9f92fc1f622-host-kubelet\") pod \"ovnkube-node-bjc8c\" (UID: \"224cde93-d872-474d-b1c4-c9f92fc1f622\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" Jul 23 00:08:08.776107 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.774446 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/224cde93-d872-474d-b1c4-c9f92fc1f622-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bjc8c\" (UID: \"224cde93-d872-474d-b1c4-c9f92fc1f622\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" Jul 23 00:08:08.776107 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.774488 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/224cde93-d872-474d-b1c4-c9f92fc1f622-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bjc8c\" (UID: \"224cde93-d872-474d-b1c4-c9f92fc1f622\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" Jul 23 00:08:08.776107 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.774487 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/216d84cd-f540-4bc0-9bd1-a8cc9e37367f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8ghxf\" (UID: \"216d84cd-f540-4bc0-9bd1-a8cc9e37367f\") " pod="openshift-multus/multus-additional-cni-plugins-8ghxf" Jul 23 00:08:08.776107 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.774521 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f9853266-a585-469c-885e-79cc72f88d18-host-slash\") pod \"iptables-alerter-626nn\" (UID: \"f9853266-a585-469c-885e-79cc72f88d18\") " pod="openshift-network-operator/iptables-alerter-626nn" Jul 23 00:08:08.776107 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.774523 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/224cde93-d872-474d-b1c4-c9f92fc1f622-host-kubelet\") pod \"ovnkube-node-bjc8c\" (UID: \"224cde93-d872-474d-b1c4-c9f92fc1f622\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" Jul 23 00:08:08.776107 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.774546 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/216d84cd-f540-4bc0-9bd1-a8cc9e37367f-system-cni-dir\") pod \"multus-additional-cni-plugins-8ghxf\" (UID: \"216d84cd-f540-4bc0-9bd1-a8cc9e37367f\") " pod="openshift-multus/multus-additional-cni-plugins-8ghxf" Jul 23 00:08:08.776107 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.774561 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f9853266-a585-469c-885e-79cc72f88d18-host-slash\") pod \"iptables-alerter-626nn\" (UID: \"f9853266-a585-469c-885e-79cc72f88d18\") " pod="openshift-network-operator/iptables-alerter-626nn" Jul 23 00:08:08.776107 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.774575 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb94m\" (UniqueName: \"kubernetes.io/projected/3749b807-a46e-43b6-880a-ac9bec9200f6-kube-api-access-hb94m\") pod \"aws-ebs-csi-driver-node-mvpdz\" (UID: \"3749b807-a46e-43b6-880a-ac9bec9200f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvpdz" Jul 23 00:08:08.776107 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.774584 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/216d84cd-f540-4bc0-9bd1-a8cc9e37367f-system-cni-dir\") pod \"multus-additional-cni-plugins-8ghxf\" (UID: \"216d84cd-f540-4bc0-9bd1-a8cc9e37367f\") " pod="openshift-multus/multus-additional-cni-plugins-8ghxf" Jul 23 00:08:08.776107 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.774607 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/216d84cd-f540-4bc0-9bd1-a8cc9e37367f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8ghxf\" (UID: \"216d84cd-f540-4bc0-9bd1-a8cc9e37367f\") " pod="openshift-multus/multus-additional-cni-plugins-8ghxf" Jul 23 00:08:08.776107 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.774635 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9b166998-6f59-4909-b0aa-4c015ce61565-multus-daemon-config\") pod \"multus-vgrs2\" (UID: \"9b166998-6f59-4909-b0aa-4c015ce61565\") " pod="openshift-multus/multus-vgrs2" Jul 23 00:08:08.776107 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.774660 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5f272157-a81d-439c-86ed-511e11146698-kubelet-config\") pod \"global-pull-secret-syncer-jlsd6\" (UID: \"5f272157-a81d-439c-86ed-511e11146698\") " pod="kube-system/global-pull-secret-syncer-jlsd6" Jul 23 00:08:08.776107 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.774700 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/af5c6a95-1825-4855-9efb-906c85d29aa5-var-lib-kubelet\") pod \"tuned-sr9sh\" (UID: \"af5c6a95-1825-4855-9efb-906c85d29aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9sh" Jul 23 00:08:08.776107 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.774713 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5f272157-a81d-439c-86ed-511e11146698-kubelet-config\") pod \"global-pull-secret-syncer-jlsd6\" (UID: \"5f272157-a81d-439c-86ed-511e11146698\") " pod="kube-system/global-pull-secret-syncer-jlsd6" Jul 23 00:08:08.777315 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.774727 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bts77\" (UniqueName: \"kubernetes.io/projected/af5c6a95-1825-4855-9efb-906c85d29aa5-kube-api-access-bts77\") pod \"tuned-sr9sh\" (UID: \"af5c6a95-1825-4855-9efb-906c85d29aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9sh" Jul 23 00:08:08.777315 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.774750 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/224cde93-d872-474d-b1c4-c9f92fc1f622-host-slash\") pod \"ovnkube-node-bjc8c\" (UID: \"224cde93-d872-474d-b1c4-c9f92fc1f622\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" Jul 23 00:08:08.777315 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.774777 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/af5c6a95-1825-4855-9efb-906c85d29aa5-var-lib-kubelet\") pod \"tuned-sr9sh\" (UID: \"af5c6a95-1825-4855-9efb-906c85d29aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9sh" Jul 23 00:08:08.777315 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.774789 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/224cde93-d872-474d-b1c4-c9f92fc1f622-etc-openvswitch\") pod \"ovnkube-node-bjc8c\" (UID: \"224cde93-d872-474d-b1c4-c9f92fc1f622\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" Jul 23 00:08:08.777315 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.774815 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/224cde93-d872-474d-b1c4-c9f92fc1f622-host-slash\") pod \"ovnkube-node-bjc8c\" (UID: \"224cde93-d872-474d-b1c4-c9f92fc1f622\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" Jul 23 00:08:08.777315 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.774814 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3749b807-a46e-43b6-880a-ac9bec9200f6-registration-dir\") pod \"aws-ebs-csi-driver-node-mvpdz\" (UID: \"3749b807-a46e-43b6-880a-ac9bec9200f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvpdz" Jul 23 00:08:08.777315 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.774853 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/af5c6a95-1825-4855-9efb-906c85d29aa5-host\") pod \"tuned-sr9sh\" (UID: \"af5c6a95-1825-4855-9efb-906c85d29aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9sh" Jul 23 00:08:08.777315 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.774852 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/224cde93-d872-474d-b1c4-c9f92fc1f622-etc-openvswitch\") pod \"ovnkube-node-bjc8c\" (UID: \"224cde93-d872-474d-b1c4-c9f92fc1f622\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" Jul 23 00:08:08.777315 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.774877 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af5c6a95-1825-4855-9efb-906c85d29aa5-tmp\") pod \"tuned-sr9sh\" (UID: \"af5c6a95-1825-4855-9efb-906c85d29aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9sh" Jul 23 00:08:08.777315 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.774915 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/224cde93-d872-474d-b1c4-c9f92fc1f622-systemd-units\") pod \"ovnkube-node-bjc8c\" (UID: \"224cde93-d872-474d-b1c4-c9f92fc1f622\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" Jul 23 00:08:08.777315 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.774938 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/224cde93-d872-474d-b1c4-c9f92fc1f622-host-cni-bin\") pod \"ovnkube-node-bjc8c\" (UID: \"224cde93-d872-474d-b1c4-c9f92fc1f622\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" Jul 23 00:08:08.777315 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.774963 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/224cde93-d872-474d-b1c4-c9f92fc1f622-ovn-node-metrics-cert\") pod \"ovnkube-node-bjc8c\" (UID: \"224cde93-d872-474d-b1c4-c9f92fc1f622\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" Jul 23 00:08:08.777315 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.774988 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwcm6\" (UniqueName: \"kubernetes.io/projected/5a9bdee1-a493-4223-a7b6-3ae1f5024eb6-kube-api-access-kwcm6\") pod \"network-metrics-daemon-kfgzz\" (UID: \"5a9bdee1-a493-4223-a7b6-3ae1f5024eb6\") " pod="openshift-multus/network-metrics-daemon-kfgzz" Jul 23 00:08:08.777315 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.775013 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f9853266-a585-469c-885e-79cc72f88d18-iptables-alerter-script\") pod \"iptables-alerter-626nn\" (UID: \"f9853266-a585-469c-885e-79cc72f88d18\") " pod="openshift-network-operator/iptables-alerter-626nn" Jul 23 00:08:08.777996 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.775032 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/224cde93-d872-474d-b1c4-c9f92fc1f622-host-cni-bin\") pod \"ovnkube-node-bjc8c\" (UID: \"224cde93-d872-474d-b1c4-c9f92fc1f622\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" Jul 23 00:08:08.777996 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.775039 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/aeaeb6df-4c50-464f-b5fa-a6e599d2fdf9-serviceca\") pod \"node-ca-jctrc\" (UID: \"aeaeb6df-4c50-464f-b5fa-a6e599d2fdf9\") " pod="openshift-image-registry/node-ca-jctrc" Jul 23 00:08:08.777996 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.775244 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/216d84cd-f540-4bc0-9bd1-a8cc9e37367f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8ghxf\" (UID: \"216d84cd-f540-4bc0-9bd1-a8cc9e37367f\") " pod="openshift-multus/multus-additional-cni-plugins-8ghxf" Jul 23 00:08:08.777996 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.775298 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/af5c6a95-1825-4855-9efb-906c85d29aa5-run\") pod \"tuned-sr9sh\" (UID: \"af5c6a95-1825-4855-9efb-906c85d29aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9sh" Jul 23 00:08:08.777996 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.775391 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/224cde93-d872-474d-b1c4-c9f92fc1f622-log-socket\") pod \"ovnkube-node-bjc8c\" (UID: \"224cde93-d872-474d-b1c4-c9f92fc1f622\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" Jul 23 00:08:08.777996 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.775423 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/af5c6a95-1825-4855-9efb-906c85d29aa5-run\") pod \"tuned-sr9sh\" (UID: \"af5c6a95-1825-4855-9efb-906c85d29aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9sh" Jul 23 00:08:08.777996 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.775436 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzxdr\" (UniqueName: \"kubernetes.io/projected/f9853266-a585-469c-885e-79cc72f88d18-kube-api-access-lzxdr\") pod \"iptables-alerter-626nn\" (UID: \"f9853266-a585-469c-885e-79cc72f88d18\") " pod="openshift-network-operator/iptables-alerter-626nn" Jul 23 00:08:08.777996 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.775438 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/af5c6a95-1825-4855-9efb-906c85d29aa5-host\") pod \"tuned-sr9sh\" (UID: \"af5c6a95-1825-4855-9efb-906c85d29aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9sh" Jul 23 00:08:08.777996 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.775463 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/216d84cd-f540-4bc0-9bd1-a8cc9e37367f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8ghxf\" (UID: \"216d84cd-f540-4bc0-9bd1-a8cc9e37367f\") " pod="openshift-multus/multus-additional-cni-plugins-8ghxf" Jul 23 00:08:08.777996 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.775491 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f272157-a81d-439c-86ed-511e11146698-original-pull-secret\") pod \"global-pull-secret-syncer-jlsd6\" (UID: \"5f272157-a81d-439c-86ed-511e11146698\") " pod="kube-system/global-pull-secret-syncer-jlsd6" Jul 23 00:08:08.777996 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.775516 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/19f5db96-5df5-4b0c-8cc1-89d3f2719ed2-agent-certs\") pod \"konnectivity-agent-fhbjd\" (UID: \"19f5db96-5df5-4b0c-8cc1-89d3f2719ed2\") " pod="kube-system/konnectivity-agent-fhbjd" Jul 23 00:08:08.777996 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.775541 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9b166998-6f59-4909-b0aa-4c015ce61565-host-run-k8s-cni-cncf-io\") pod \"multus-vgrs2\" (UID: \"9b166998-6f59-4909-b0aa-4c015ce61565\") " pod="openshift-multus/multus-vgrs2" Jul 23 00:08:08.777996 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.775564 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/224cde93-d872-474d-b1c4-c9f92fc1f622-run-ovn\") pod \"ovnkube-node-bjc8c\" (UID: \"224cde93-d872-474d-b1c4-c9f92fc1f622\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" Jul 23 00:08:08.777996 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.775588 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/224cde93-d872-474d-b1c4-c9f92fc1f622-ovnkube-config\") pod \"ovnkube-node-bjc8c\" (UID: \"224cde93-d872-474d-b1c4-c9f92fc1f622\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" Jul 23 00:08:08.778637 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.775615 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/224cde93-d872-474d-b1c4-c9f92fc1f622-env-overrides\") pod \"ovnkube-node-bjc8c\" (UID: \"224cde93-d872-474d-b1c4-c9f92fc1f622\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" Jul 23 00:08:08.778637 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.775642 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aeaeb6df-4c50-464f-b5fa-a6e599d2fdf9-host\") pod \"node-ca-jctrc\" (UID: \"aeaeb6df-4c50-464f-b5fa-a6e599d2fdf9\") " pod="openshift-image-registry/node-ca-jctrc" Jul 23 00:08:08.778637 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.775468 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/224cde93-d872-474d-b1c4-c9f92fc1f622-log-socket\") pod \"ovnkube-node-bjc8c\" (UID: \"224cde93-d872-474d-b1c4-c9f92fc1f622\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" Jul 23 00:08:08.778637 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.775671 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/af5c6a95-1825-4855-9efb-906c85d29aa5-lib-modules\") pod \"tuned-sr9sh\" (UID: \"af5c6a95-1825-4855-9efb-906c85d29aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9sh" Jul 23 00:08:08.778637 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.775695 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a9bdee1-a493-4223-a7b6-3ae1f5024eb6-metrics-certs\") pod \"network-metrics-daemon-kfgzz\" (UID: \"5a9bdee1-a493-4223-a7b6-3ae1f5024eb6\") " pod="openshift-multus/network-metrics-daemon-kfgzz" Jul 23 00:08:08.778637 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.775719 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpdz6\" (UniqueName: \"kubernetes.io/projected/216d84cd-f540-4bc0-9bd1-a8cc9e37367f-kube-api-access-cpdz6\") pod \"multus-additional-cni-plugins-8ghxf\" (UID: \"216d84cd-f540-4bc0-9bd1-a8cc9e37367f\") " pod="openshift-multus/multus-additional-cni-plugins-8ghxf" Jul 23 00:08:08.778637 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.775735 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f9853266-a585-469c-885e-79cc72f88d18-iptables-alerter-script\") pod \"iptables-alerter-626nn\" (UID: \"f9853266-a585-469c-885e-79cc72f88d18\") " pod="openshift-network-operator/iptables-alerter-626nn" Jul 23 00:08:08.778637 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:08.775736 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 00:08:08.778637 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.775749 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctpsk\" (UniqueName: \"kubernetes.io/projected/aeaeb6df-4c50-464f-b5fa-a6e599d2fdf9-kube-api-access-ctpsk\") pod \"node-ca-jctrc\" (UID: \"aeaeb6df-4c50-464f-b5fa-a6e599d2fdf9\") " pod="openshift-image-registry/node-ca-jctrc" Jul 23 00:08:08.778637 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.775815 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9b166998-6f59-4909-b0aa-4c015ce61565-cnibin\") pod \"multus-vgrs2\" (UID: \"9b166998-6f59-4909-b0aa-4c015ce61565\") " pod="openshift-multus/multus-vgrs2" Jul 23 00:08:08.778637 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.775799 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/224cde93-d872-474d-b1c4-c9f92fc1f622-run-ovn\") pod \"ovnkube-node-bjc8c\" (UID: \"224cde93-d872-474d-b1c4-c9f92fc1f622\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" Jul 23 00:08:08.778637 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:08.775842 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f272157-a81d-439c-86ed-511e11146698-original-pull-secret podName:5f272157-a81d-439c-86ed-511e11146698 nodeName:}" failed. No retries permitted until 2026-07-23 00:08:09.275818029 +0000 UTC m=+3.071324870 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5f272157-a81d-439c-86ed-511e11146698-original-pull-secret") pod "global-pull-secret-syncer-jlsd6" (UID: "5f272157-a81d-439c-86ed-511e11146698") : object "kube-system"/"original-pull-secret" not registered Jul 23 00:08:08.778637 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.775927 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/216d84cd-f540-4bc0-9bd1-a8cc9e37367f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8ghxf\" (UID: \"216d84cd-f540-4bc0-9bd1-a8cc9e37367f\") " pod="openshift-multus/multus-additional-cni-plugins-8ghxf" Jul 23 00:08:08.778637 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.775979 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9b166998-6f59-4909-b0aa-4c015ce61565-cni-binary-copy\") pod \"multus-vgrs2\" (UID: \"9b166998-6f59-4909-b0aa-4c015ce61565\") " pod="openshift-multus/multus-vgrs2" Jul 23 00:08:08.779351 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.776005 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9b166998-6f59-4909-b0aa-4c015ce61565-host-run-netns\") pod \"multus-vgrs2\" (UID: \"9b166998-6f59-4909-b0aa-4c015ce61565\") " pod="openshift-multus/multus-vgrs2" Jul 23 00:08:08.779351 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.776029 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/af5c6a95-1825-4855-9efb-906c85d29aa5-etc-sysctl-d\") pod \"tuned-sr9sh\" (UID: \"af5c6a95-1825-4855-9efb-906c85d29aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9sh" Jul 23 00:08:08.779351 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.776138 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/af5c6a95-1825-4855-9efb-906c85d29aa5-lib-modules\") pod \"tuned-sr9sh\" (UID: \"af5c6a95-1825-4855-9efb-906c85d29aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9sh" Jul 23 00:08:08.779351 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.776176 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/216d84cd-f540-4bc0-9bd1-a8cc9e37367f-cni-binary-copy\") pod \"multus-additional-cni-plugins-8ghxf\" (UID: \"216d84cd-f540-4bc0-9bd1-a8cc9e37367f\") " pod="openshift-multus/multus-additional-cni-plugins-8ghxf" Jul 23 00:08:08.779351 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.776206 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsxgl\" (UniqueName: \"kubernetes.io/projected/7d457c50-6bca-4609-8897-cef340b087e8-kube-api-access-gsxgl\") pod \"node-resolver-9stwt\" (UID: \"7d457c50-6bca-4609-8897-cef340b087e8\") " pod="openshift-dns/node-resolver-9stwt" Jul 23 00:08:08.779351 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.776258 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/af5c6a95-1825-4855-9efb-906c85d29aa5-etc-sysctl-d\") pod \"tuned-sr9sh\" (UID: \"af5c6a95-1825-4855-9efb-906c85d29aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9sh" Jul 23 00:08:08.779351 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.776310 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/af5c6a95-1825-4855-9efb-906c85d29aa5-etc-sysconfig\") pod \"tuned-sr9sh\" (UID: \"af5c6a95-1825-4855-9efb-906c85d29aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9sh" Jul 23 00:08:08.779351 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.776316 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/224cde93-d872-474d-b1c4-c9f92fc1f622-ovnkube-config\") pod \"ovnkube-node-bjc8c\" (UID: \"224cde93-d872-474d-b1c4-c9f92fc1f622\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" Jul 23 00:08:08.779351 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:08.776178 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 00:08:08.779351 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.776263 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/af5c6a95-1825-4855-9efb-906c85d29aa5-etc-sysconfig\") pod \"tuned-sr9sh\" (UID: \"af5c6a95-1825-4855-9efb-906c85d29aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9sh" Jul 23 00:08:08.779351 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.776372 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/224cde93-d872-474d-b1c4-c9f92fc1f622-run-systemd\") pod \"ovnkube-node-bjc8c\" (UID: \"224cde93-d872-474d-b1c4-c9f92fc1f622\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" Jul 23 00:08:08.779351 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.776373 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/224cde93-d872-474d-b1c4-c9f92fc1f622-env-overrides\") pod \"ovnkube-node-bjc8c\" (UID: \"224cde93-d872-474d-b1c4-c9f92fc1f622\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" Jul 23 00:08:08.779351 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:08.776408 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a9bdee1-a493-4223-a7b6-3ae1f5024eb6-metrics-certs podName:5a9bdee1-a493-4223-a7b6-3ae1f5024eb6 nodeName:}" failed. No retries permitted until 2026-07-23 00:08:09.27636781 +0000 UTC m=+3.071874646 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5a9bdee1-a493-4223-a7b6-3ae1f5024eb6-metrics-certs") pod "network-metrics-daemon-kfgzz" (UID: "5a9bdee1-a493-4223-a7b6-3ae1f5024eb6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 00:08:08.779351 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.776413 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/224cde93-d872-474d-b1c4-c9f92fc1f622-run-systemd\") pod \"ovnkube-node-bjc8c\" (UID: \"224cde93-d872-474d-b1c4-c9f92fc1f622\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" Jul 23 00:08:08.779954 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.776468 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2thdl\" (UniqueName: \"kubernetes.io/projected/72c0b00d-f59f-4186-b84e-470a1adac941-kube-api-access-2thdl\") pod \"network-check-target-cgvgd\" (UID: \"72c0b00d-f59f-4186-b84e-470a1adac941\") " pod="openshift-network-diagnostics/network-check-target-cgvgd" Jul 23 00:08:08.779954 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.776495 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5f272157-a81d-439c-86ed-511e11146698-dbus\") pod \"global-pull-secret-syncer-jlsd6\" (UID: \"5f272157-a81d-439c-86ed-511e11146698\") " pod="kube-system/global-pull-secret-syncer-jlsd6" Jul 23 00:08:08.779954 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.776529 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/224cde93-d872-474d-b1c4-c9f92fc1f622-systemd-units\") pod \"ovnkube-node-bjc8c\" (UID: \"224cde93-d872-474d-b1c4-c9f92fc1f622\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" Jul 23 00:08:08.779954 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.776693 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5f272157-a81d-439c-86ed-511e11146698-dbus\") pod \"global-pull-secret-syncer-jlsd6\" (UID: \"5f272157-a81d-439c-86ed-511e11146698\") " pod="kube-system/global-pull-secret-syncer-jlsd6" Jul 23 00:08:08.779954 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.776720 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/216d84cd-f540-4bc0-9bd1-a8cc9e37367f-cni-binary-copy\") pod \"multus-additional-cni-plugins-8ghxf\" (UID: \"216d84cd-f540-4bc0-9bd1-a8cc9e37367f\") " pod="openshift-multus/multus-additional-cni-plugins-8ghxf" Jul 23 00:08:08.779954 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.779019 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af5c6a95-1825-4855-9efb-906c85d29aa5-tmp\") pod \"tuned-sr9sh\" (UID: \"af5c6a95-1825-4855-9efb-906c85d29aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9sh" Jul 23 00:08:08.779954 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.779077 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/af5c6a95-1825-4855-9efb-906c85d29aa5-etc-tuned\") pod \"tuned-sr9sh\" (UID: \"af5c6a95-1825-4855-9efb-906c85d29aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9sh" Jul 23 00:08:08.779954 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.779384 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/224cde93-d872-474d-b1c4-c9f92fc1f622-ovn-node-metrics-cert\") pod \"ovnkube-node-bjc8c\" (UID: \"224cde93-d872-474d-b1c4-c9f92fc1f622\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" Jul 23 00:08:08.782862 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.782832 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5x4x\" (UniqueName: \"kubernetes.io/projected/224cde93-d872-474d-b1c4-c9f92fc1f622-kube-api-access-g5x4x\") pod \"ovnkube-node-bjc8c\" (UID: \"224cde93-d872-474d-b1c4-c9f92fc1f622\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" Jul 23 00:08:08.783993 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.783851 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bts77\" (UniqueName: \"kubernetes.io/projected/af5c6a95-1825-4855-9efb-906c85d29aa5-kube-api-access-bts77\") pod \"tuned-sr9sh\" (UID: \"af5c6a95-1825-4855-9efb-906c85d29aa5\") " pod="openshift-cluster-node-tuning-operator/tuned-sr9sh" Jul 23 00:08:08.784390 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.784367 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzxdr\" (UniqueName: \"kubernetes.io/projected/f9853266-a585-469c-885e-79cc72f88d18-kube-api-access-lzxdr\") pod \"iptables-alerter-626nn\" (UID: \"f9853266-a585-469c-885e-79cc72f88d18\") " pod="openshift-network-operator/iptables-alerter-626nn" Jul 23 00:08:08.785448 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.785423 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwcm6\" (UniqueName: \"kubernetes.io/projected/5a9bdee1-a493-4223-a7b6-3ae1f5024eb6-kube-api-access-kwcm6\") pod \"network-metrics-daemon-kfgzz\" (UID: \"5a9bdee1-a493-4223-a7b6-3ae1f5024eb6\") " pod="openshift-multus/network-metrics-daemon-kfgzz" Jul 23 00:08:08.786588 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:08.786551 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 00:08:08.786588 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:08.786581 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 00:08:08.786738 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:08.786597 2562 projected.go:196] Error preparing data for projected volume kube-api-access-2thdl for pod openshift-network-diagnostics/network-check-target-cgvgd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 00:08:08.786738 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:08.786664 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/72c0b00d-f59f-4186-b84e-470a1adac941-kube-api-access-2thdl podName:72c0b00d-f59f-4186-b84e-470a1adac941 nodeName:}" failed. No retries permitted until 2026-07-23 00:08:09.286645249 +0000 UTC m=+3.082152090 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2thdl" (UniqueName: "kubernetes.io/projected/72c0b00d-f59f-4186-b84e-470a1adac941-kube-api-access-2thdl") pod "network-check-target-cgvgd" (UID: "72c0b00d-f59f-4186-b84e-470a1adac941") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 00:08:08.787685 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.787645 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpdz6\" (UniqueName: \"kubernetes.io/projected/216d84cd-f540-4bc0-9bd1-a8cc9e37367f-kube-api-access-cpdz6\") pod \"multus-additional-cni-plugins-8ghxf\" (UID: \"216d84cd-f540-4bc0-9bd1-a8cc9e37367f\") " pod="openshift-multus/multus-additional-cni-plugins-8ghxf" Jul 23 00:08:08.809912 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.809876 2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 00:08:08.877996 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.877234 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3749b807-a46e-43b6-880a-ac9bec9200f6-etc-selinux\") pod \"aws-ebs-csi-driver-node-mvpdz\" (UID: \"3749b807-a46e-43b6-880a-ac9bec9200f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvpdz" Jul 23 00:08:08.877996 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.877336 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b166998-6f59-4909-b0aa-4c015ce61565-etc-kubernetes\") pod \"multus-vgrs2\" (UID: \"9b166998-6f59-4909-b0aa-4c015ce61565\") " pod="openshift-multus/multus-vgrs2" Jul 23 00:08:08.877996 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.877369 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b166998-6f59-4909-b0aa-4c015ce61565-system-cni-dir\") pod \"multus-vgrs2\" (UID: \"9b166998-6f59-4909-b0aa-4c015ce61565\") " pod="openshift-multus/multus-vgrs2" Jul 23 00:08:08.877996 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.877398 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b166998-6f59-4909-b0aa-4c015ce61565-multus-cni-dir\") pod \"multus-vgrs2\" (UID: \"9b166998-6f59-4909-b0aa-4c015ce61565\") " pod="openshift-multus/multus-vgrs2" Jul 23 00:08:08.877996 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.877447 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9b166998-6f59-4909-b0aa-4c015ce61565-host-var-lib-cni-bin\") pod \"multus-vgrs2\" (UID: \"9b166998-6f59-4909-b0aa-4c015ce61565\") " pod="openshift-multus/multus-vgrs2" Jul 23 00:08:08.877996 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.877474 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9b166998-6f59-4909-b0aa-4c015ce61565-hostroot\") pod \"multus-vgrs2\" (UID: \"9b166998-6f59-4909-b0aa-4c015ce61565\") " pod="openshift-multus/multus-vgrs2" Jul 23 00:08:08.877996 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.877498 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b166998-6f59-4909-b0aa-4c015ce61565-system-cni-dir\") pod \"multus-vgrs2\" (UID: \"9b166998-6f59-4909-b0aa-4c015ce61565\") " pod="openshift-multus/multus-vgrs2" Jul 23 00:08:08.877996 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.877558 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b166998-6f59-4909-b0aa-4c015ce61565-etc-kubernetes\") pod \"multus-vgrs2\" (UID: \"9b166998-6f59-4909-b0aa-4c015ce61565\") " pod="openshift-multus/multus-vgrs2" Jul 23 00:08:08.877996 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.877498 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9b166998-6f59-4909-b0aa-4c015ce61565-multus-conf-dir\") pod \"multus-vgrs2\" (UID: \"9b166998-6f59-4909-b0aa-4c015ce61565\") " pod="openshift-multus/multus-vgrs2" Jul 23 00:08:08.877996 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.877600 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9b166998-6f59-4909-b0aa-4c015ce61565-host-var-lib-cni-bin\") pod \"multus-vgrs2\" (UID: \"9b166998-6f59-4909-b0aa-4c015ce61565\") " pod="openshift-multus/multus-vgrs2" Jul 23 00:08:08.877996 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.877604 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3749b807-a46e-43b6-880a-ac9bec9200f6-sys-fs\") pod \"aws-ebs-csi-driver-node-mvpdz\" (UID: \"3749b807-a46e-43b6-880a-ac9bec9200f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvpdz" Jul 23 00:08:08.877996 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.877633 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9b166998-6f59-4909-b0aa-4c015ce61565-host-run-multus-certs\") pod \"multus-vgrs2\" (UID: \"9b166998-6f59-4909-b0aa-4c015ce61565\") " pod="openshift-multus/multus-vgrs2" Jul 23 00:08:08.877996 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.877655 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b166998-6f59-4909-b0aa-4c015ce61565-multus-cni-dir\") pod \"multus-vgrs2\" (UID: \"9b166998-6f59-4909-b0aa-4c015ce61565\") " pod="openshift-multus/multus-vgrs2" Jul 23 00:08:08.877996 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.877661 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb94m\" (UniqueName: \"kubernetes.io/projected/3749b807-a46e-43b6-880a-ac9bec9200f6-kube-api-access-hb94m\") pod \"aws-ebs-csi-driver-node-mvpdz\" (UID: \"3749b807-a46e-43b6-880a-ac9bec9200f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvpdz" Jul 23 00:08:08.877996 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.877686 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9b166998-6f59-4909-b0aa-4c015ce61565-multus-daemon-config\") pod \"multus-vgrs2\" (UID: \"9b166998-6f59-4909-b0aa-4c015ce61565\") " pod="openshift-multus/multus-vgrs2" Jul 23 00:08:08.878897 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.877714 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9b166998-6f59-4909-b0aa-4c015ce61565-multus-conf-dir\") pod \"multus-vgrs2\" (UID: \"9b166998-6f59-4909-b0aa-4c015ce61565\") " pod="openshift-multus/multus-vgrs2" Jul 23 00:08:08.878897 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.877446 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3749b807-a46e-43b6-880a-ac9bec9200f6-etc-selinux\") pod \"aws-ebs-csi-driver-node-mvpdz\" (UID: \"3749b807-a46e-43b6-880a-ac9bec9200f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvpdz" Jul 23 00:08:08.878897 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.877717 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3749b807-a46e-43b6-880a-ac9bec9200f6-registration-dir\") pod \"aws-ebs-csi-driver-node-mvpdz\" (UID: \"3749b807-a46e-43b6-880a-ac9bec9200f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvpdz" Jul 23 00:08:08.878897 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.877788 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/aeaeb6df-4c50-464f-b5fa-a6e599d2fdf9-serviceca\") pod \"node-ca-jctrc\" (UID: \"aeaeb6df-4c50-464f-b5fa-a6e599d2fdf9\") " pod="openshift-image-registry/node-ca-jctrc" Jul 23 00:08:08.878897 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.877800 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3749b807-a46e-43b6-880a-ac9bec9200f6-registration-dir\") pod \"aws-ebs-csi-driver-node-mvpdz\" (UID: \"3749b807-a46e-43b6-880a-ac9bec9200f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvpdz" Jul 23 00:08:08.878897 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.877835 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/19f5db96-5df5-4b0c-8cc1-89d3f2719ed2-agent-certs\") pod \"konnectivity-agent-fhbjd\" (UID: \"19f5db96-5df5-4b0c-8cc1-89d3f2719ed2\") " pod="kube-system/konnectivity-agent-fhbjd" Jul 23 00:08:08.878897 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.877863 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9b166998-6f59-4909-b0aa-4c015ce61565-host-run-k8s-cni-cncf-io\") pod \"multus-vgrs2\" (UID: \"9b166998-6f59-4909-b0aa-4c015ce61565\") " pod="openshift-multus/multus-vgrs2" Jul 23 00:08:08.878897 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.877873 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3749b807-a46e-43b6-880a-ac9bec9200f6-sys-fs\") pod \"aws-ebs-csi-driver-node-mvpdz\" (UID: \"3749b807-a46e-43b6-880a-ac9bec9200f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvpdz" Jul 23 00:08:08.878897 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.877890 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aeaeb6df-4c50-464f-b5fa-a6e599d2fdf9-host\") pod \"node-ca-jctrc\" (UID: \"aeaeb6df-4c50-464f-b5fa-a6e599d2fdf9\") " pod="openshift-image-registry/node-ca-jctrc" Jul 23 00:08:08.878897 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.877934 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctpsk\" (UniqueName: \"kubernetes.io/projected/aeaeb6df-4c50-464f-b5fa-a6e599d2fdf9-kube-api-access-ctpsk\") pod \"node-ca-jctrc\" (UID: \"aeaeb6df-4c50-464f-b5fa-a6e599d2fdf9\") " pod="openshift-image-registry/node-ca-jctrc" Jul 23 00:08:08.878897 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.877959 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9b166998-6f59-4909-b0aa-4c015ce61565-cnibin\") pod \"multus-vgrs2\" (UID: \"9b166998-6f59-4909-b0aa-4c015ce61565\") " pod="openshift-multus/multus-vgrs2" Jul 23 00:08:08.878897 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.877983 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9b166998-6f59-4909-b0aa-4c015ce61565-cni-binary-copy\") pod \"multus-vgrs2\" (UID: \"9b166998-6f59-4909-b0aa-4c015ce61565\") " pod="openshift-multus/multus-vgrs2" Jul 23 00:08:08.878897 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.878004 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9b166998-6f59-4909-b0aa-4c015ce61565-host-run-netns\") pod \"multus-vgrs2\" (UID: \"9b166998-6f59-4909-b0aa-4c015ce61565\") " pod="openshift-multus/multus-vgrs2" Jul 23 00:08:08.878897 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.878028 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsxgl\" (UniqueName: \"kubernetes.io/projected/7d457c50-6bca-4609-8897-cef340b087e8-kube-api-access-gsxgl\") pod \"node-resolver-9stwt\" (UID: \"7d457c50-6bca-4609-8897-cef340b087e8\") " pod="openshift-dns/node-resolver-9stwt" Jul 23 00:08:08.878897 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.878064 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d457c50-6bca-4609-8897-cef340b087e8-tmp-dir\") pod \"node-resolver-9stwt\" (UID: \"7d457c50-6bca-4609-8897-cef340b087e8\") " pod="openshift-dns/node-resolver-9stwt" Jul 23 00:08:08.879675 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.878084 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3749b807-a46e-43b6-880a-ac9bec9200f6-device-dir\") pod \"aws-ebs-csi-driver-node-mvpdz\" (UID: \"3749b807-a46e-43b6-880a-ac9bec9200f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvpdz" Jul 23 00:08:08.879675 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.878105 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9b166998-6f59-4909-b0aa-4c015ce61565-os-release\") pod \"multus-vgrs2\" (UID: \"9b166998-6f59-4909-b0aa-4c015ce61565\") " pod="openshift-multus/multus-vgrs2" Jul 23 00:08:08.879675 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.878125 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9b166998-6f59-4909-b0aa-4c015ce61565-host-var-lib-cni-multus\") pod \"multus-vgrs2\" (UID: \"9b166998-6f59-4909-b0aa-4c015ce61565\") " pod="openshift-multus/multus-vgrs2" Jul 23 00:08:08.879675 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.878150 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7d457c50-6bca-4609-8897-cef340b087e8-hosts-file\") pod \"node-resolver-9stwt\" (UID: \"7d457c50-6bca-4609-8897-cef340b087e8\") " pod="openshift-dns/node-resolver-9stwt" Jul 23 00:08:08.879675 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.878174 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/19f5db96-5df5-4b0c-8cc1-89d3f2719ed2-konnectivity-ca\") pod \"konnectivity-agent-fhbjd\" (UID: \"19f5db96-5df5-4b0c-8cc1-89d3f2719ed2\") " pod="kube-system/konnectivity-agent-fhbjd" Jul 23 00:08:08.879675 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.878195 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3749b807-a46e-43b6-880a-ac9bec9200f6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mvpdz\" (UID: \"3749b807-a46e-43b6-880a-ac9bec9200f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvpdz" Jul 23 00:08:08.879675 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.877983 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9b166998-6f59-4909-b0aa-4c015ce61565-hostroot\") pod \"multus-vgrs2\" (UID: \"9b166998-6f59-4909-b0aa-4c015ce61565\") " pod="openshift-multus/multus-vgrs2" Jul 23 00:08:08.879675 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.878380 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/aeaeb6df-4c50-464f-b5fa-a6e599d2fdf9-serviceca\") pod \"node-ca-jctrc\" (UID: \"aeaeb6df-4c50-464f-b5fa-a6e599d2fdf9\") " pod="openshift-image-registry/node-ca-jctrc" Jul 23 00:08:08.879675 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.878438 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9b166998-6f59-4909-b0aa-4c015ce61565-os-release\") pod \"multus-vgrs2\" (UID: \"9b166998-6f59-4909-b0aa-4c015ce61565\") " pod="openshift-multus/multus-vgrs2" Jul 23 00:08:08.879675 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.878463 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9b166998-6f59-4909-b0aa-4c015ce61565-host-run-netns\") pod \"multus-vgrs2\" (UID: \"9b166998-6f59-4909-b0aa-4c015ce61565\") " pod="openshift-multus/multus-vgrs2" Jul 23 00:08:08.879675 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.878466 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9b166998-6f59-4909-b0aa-4c015ce61565-multus-daemon-config\") pod \"multus-vgrs2\" (UID: \"9b166998-6f59-4909-b0aa-4c015ce61565\") " pod="openshift-multus/multus-vgrs2" Jul 23 00:08:08.879675 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.878533 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aeaeb6df-4c50-464f-b5fa-a6e599d2fdf9-host\") pod \"node-ca-jctrc\" (UID: \"aeaeb6df-4c50-464f-b5fa-a6e599d2fdf9\") " pod="openshift-image-registry/node-ca-jctrc" Jul 23 00:08:08.879675 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.878582 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3749b807-a46e-43b6-880a-ac9bec9200f6-device-dir\") pod \"aws-ebs-csi-driver-node-mvpdz\" (UID: \"3749b807-a46e-43b6-880a-ac9bec9200f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvpdz" Jul 23 00:08:08.879675 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.878629 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7d457c50-6bca-4609-8897-cef340b087e8-hosts-file\") pod \"node-resolver-9stwt\" (UID: \"7d457c50-6bca-4609-8897-cef340b087e8\") " pod="openshift-dns/node-resolver-9stwt" Jul 23 00:08:08.879675 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.878627 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9b166998-6f59-4909-b0aa-4c015ce61565-host-var-lib-cni-multus\") pod \"multus-vgrs2\" (UID: \"9b166998-6f59-4909-b0aa-4c015ce61565\") " pod="openshift-multus/multus-vgrs2" Jul 23 00:08:08.880393 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.878664 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9b166998-6f59-4909-b0aa-4c015ce61565-cnibin\") pod \"multus-vgrs2\" (UID: \"9b166998-6f59-4909-b0aa-4c015ce61565\") " pod="openshift-multus/multus-vgrs2" Jul 23 00:08:08.880393 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.877934 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9b166998-6f59-4909-b0aa-4c015ce61565-host-run-multus-certs\") pod \"multus-vgrs2\" (UID: \"9b166998-6f59-4909-b0aa-4c015ce61565\") " pod="openshift-multus/multus-vgrs2" Jul 23 00:08:08.880393 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.878691 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3749b807-a46e-43b6-880a-ac9bec9200f6-socket-dir\") pod \"aws-ebs-csi-driver-node-mvpdz\" (UID: \"3749b807-a46e-43b6-880a-ac9bec9200f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvpdz" Jul 23 00:08:08.880393 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.878727 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b166998-6f59-4909-b0aa-4c015ce61565-host-var-lib-kubelet\") pod \"multus-vgrs2\" (UID: \"9b166998-6f59-4909-b0aa-4c015ce61565\") " pod="openshift-multus/multus-vgrs2" Jul 23 00:08:08.880393 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.880095 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/19f5db96-5df5-4b0c-8cc1-89d3f2719ed2-konnectivity-ca\") pod \"konnectivity-agent-fhbjd\" (UID: \"19f5db96-5df5-4b0c-8cc1-89d3f2719ed2\") " pod="kube-system/konnectivity-agent-fhbjd" Jul 23 00:08:08.880393 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.880117 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9b166998-6f59-4909-b0aa-4c015ce61565-host-run-k8s-cni-cncf-io\") pod \"multus-vgrs2\" (UID: \"9b166998-6f59-4909-b0aa-4c015ce61565\") " pod="openshift-multus/multus-vgrs2" Jul 23 00:08:08.880393 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.880015 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d457c50-6bca-4609-8897-cef340b087e8-tmp-dir\") pod \"node-resolver-9stwt\" (UID: \"7d457c50-6bca-4609-8897-cef340b087e8\") " pod="openshift-dns/node-resolver-9stwt" Jul 23 00:08:08.880393 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.880051 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3749b807-a46e-43b6-880a-ac9bec9200f6-socket-dir\") pod \"aws-ebs-csi-driver-node-mvpdz\" (UID: \"3749b807-a46e-43b6-880a-ac9bec9200f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvpdz" Jul 23 00:08:08.880393 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.879665 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b166998-6f59-4909-b0aa-4c015ce61565-host-var-lib-kubelet\") pod \"multus-vgrs2\" (UID: \"9b166998-6f59-4909-b0aa-4c015ce61565\") " pod="openshift-multus/multus-vgrs2" Jul 23 00:08:08.880393 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.880158 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9b166998-6f59-4909-b0aa-4c015ce61565-cni-binary-copy\") pod \"multus-vgrs2\" (UID: \"9b166998-6f59-4909-b0aa-4c015ce61565\") " pod="openshift-multus/multus-vgrs2" Jul 23 00:08:08.880393 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.880180 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3749b807-a46e-43b6-880a-ac9bec9200f6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mvpdz\" (UID: \"3749b807-a46e-43b6-880a-ac9bec9200f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvpdz" Jul 23 00:08:08.880393 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.880192 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9b166998-6f59-4909-b0aa-4c015ce61565-multus-socket-dir-parent\") pod \"multus-vgrs2\" (UID: \"9b166998-6f59-4909-b0aa-4c015ce61565\") " pod="openshift-multus/multus-vgrs2" Jul 23 00:08:08.880393 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.880232 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz4qf\" (UniqueName: \"kubernetes.io/projected/9b166998-6f59-4909-b0aa-4c015ce61565-kube-api-access-bz4qf\") pod \"multus-vgrs2\" (UID: \"9b166998-6f59-4909-b0aa-4c015ce61565\") " pod="openshift-multus/multus-vgrs2" Jul 23 00:08:08.880393 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.880315 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9b166998-6f59-4909-b0aa-4c015ce61565-multus-socket-dir-parent\") pod \"multus-vgrs2\" (UID: \"9b166998-6f59-4909-b0aa-4c015ce61565\") " pod="openshift-multus/multus-vgrs2" Jul 23 00:08:08.881168 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.881147 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/19f5db96-5df5-4b0c-8cc1-89d3f2719ed2-agent-certs\") pod \"konnectivity-agent-fhbjd\" (UID: \"19f5db96-5df5-4b0c-8cc1-89d3f2719ed2\") " pod="kube-system/konnectivity-agent-fhbjd" Jul 23 00:08:08.886946 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.886910 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb94m\" (UniqueName: \"kubernetes.io/projected/3749b807-a46e-43b6-880a-ac9bec9200f6-kube-api-access-hb94m\") pod \"aws-ebs-csi-driver-node-mvpdz\" (UID: \"3749b807-a46e-43b6-880a-ac9bec9200f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvpdz" Jul 23 00:08:08.887626 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.887592 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsxgl\" (UniqueName: \"kubernetes.io/projected/7d457c50-6bca-4609-8897-cef340b087e8-kube-api-access-gsxgl\") pod \"node-resolver-9stwt\" (UID: \"7d457c50-6bca-4609-8897-cef340b087e8\") " pod="openshift-dns/node-resolver-9stwt" Jul 23 00:08:08.889039 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.889015 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz4qf\" (UniqueName: \"kubernetes.io/projected/9b166998-6f59-4909-b0aa-4c015ce61565-kube-api-access-bz4qf\") pod \"multus-vgrs2\" (UID: \"9b166998-6f59-4909-b0aa-4c015ce61565\") " pod="openshift-multus/multus-vgrs2" Jul 23 00:08:08.889992 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.889922 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctpsk\" (UniqueName: \"kubernetes.io/projected/aeaeb6df-4c50-464f-b5fa-a6e599d2fdf9-kube-api-access-ctpsk\") pod \"node-ca-jctrc\" (UID: \"aeaeb6df-4c50-464f-b5fa-a6e599d2fdf9\") " pod="openshift-image-registry/node-ca-jctrc" Jul 23 00:08:08.962379 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.962333 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-sr9sh" Jul 23 00:08:08.971705 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.971676 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8ghxf" Jul 23 00:08:08.983200 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.983167 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" Jul 23 00:08:08.995505 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:08.995468 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-626nn" Jul 23 00:08:09.005856 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:09.005793 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-121.ec2.internal" event={"ID":"19e11d045e1035a525da0259bed898ce","Type":"ContainerStarted","Data":"237d330bb1f91846b8ba7dc06f544b1d29c48d77c5d49d51b2e2305df3c5d78e"} Jul 23 00:08:09.006386 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:09.006363 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fhbjd" Jul 23 00:08:09.007230 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:09.007201 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-121.ec2.internal" event={"ID":"2dc6c03e0c457691dab513c68ebb04f1","Type":"ContainerStarted","Data":"d457c572e3527be958bee3d3cea9b19f1566feca3f504369ad1ffa3f74826d85"} Jul 23 00:08:09.014274 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:09.014225 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvpdz" Jul 23 00:08:09.023662 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:09.023634 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9stwt" Jul 23 00:08:09.029636 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:09.029602 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jctrc" Jul 23 00:08:09.037916 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:09.037880 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vgrs2" Jul 23 00:08:09.282822 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:09.282695 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f272157-a81d-439c-86ed-511e11146698-original-pull-secret\") pod \"global-pull-secret-syncer-jlsd6\" (UID: \"5f272157-a81d-439c-86ed-511e11146698\") " pod="kube-system/global-pull-secret-syncer-jlsd6" Jul 23 00:08:09.282822 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:09.282769 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a9bdee1-a493-4223-a7b6-3ae1f5024eb6-metrics-certs\") pod \"network-metrics-daemon-kfgzz\" (UID: \"5a9bdee1-a493-4223-a7b6-3ae1f5024eb6\") " pod="openshift-multus/network-metrics-daemon-kfgzz" Jul 23 00:08:09.283020 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:09.282868 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 00:08:09.283020 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:09.282936 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 00:08:09.283020 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:09.282943 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f272157-a81d-439c-86ed-511e11146698-original-pull-secret podName:5f272157-a81d-439c-86ed-511e11146698 nodeName:}" failed. No retries permitted until 2026-07-23 00:08:10.282923378 +0000 UTC m=+4.078430228 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5f272157-a81d-439c-86ed-511e11146698-original-pull-secret") pod "global-pull-secret-syncer-jlsd6" (UID: "5f272157-a81d-439c-86ed-511e11146698") : object "kube-system"/"original-pull-secret" not registered Jul 23 00:08:09.283020 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:09.282995 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a9bdee1-a493-4223-a7b6-3ae1f5024eb6-metrics-certs podName:5a9bdee1-a493-4223-a7b6-3ae1f5024eb6 nodeName:}" failed. No retries permitted until 2026-07-23 00:08:10.282983375 +0000 UTC m=+4.078490213 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5a9bdee1-a493-4223-a7b6-3ae1f5024eb6-metrics-certs") pod "network-metrics-daemon-kfgzz" (UID: "5a9bdee1-a493-4223-a7b6-3ae1f5024eb6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 00:08:09.383560 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:09.383523 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2thdl\" (UniqueName: \"kubernetes.io/projected/72c0b00d-f59f-4186-b84e-470a1adac941-kube-api-access-2thdl\") pod \"network-check-target-cgvgd\" (UID: \"72c0b00d-f59f-4186-b84e-470a1adac941\") " pod="openshift-network-diagnostics/network-check-target-cgvgd" Jul 23 00:08:09.383786 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:09.383721 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 00:08:09.383786 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:09.383747 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 00:08:09.383786 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:09.383777 2562 projected.go:196] Error preparing data for projected volume kube-api-access-2thdl for pod openshift-network-diagnostics/network-check-target-cgvgd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 00:08:09.383954 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:09.383844 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/72c0b00d-f59f-4186-b84e-470a1adac941-kube-api-access-2thdl podName:72c0b00d-f59f-4186-b84e-470a1adac941 nodeName:}" failed. No retries permitted until 2026-07-23 00:08:10.38382374 +0000 UTC m=+4.179330590 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2thdl" (UniqueName: "kubernetes.io/projected/72c0b00d-f59f-4186-b84e-470a1adac941-kube-api-access-2thdl") pod "network-check-target-cgvgd" (UID: "72c0b00d-f59f-4186-b84e-470a1adac941") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 00:08:09.701012 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:09.700902 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 00:03:08 +0000 UTC" deadline="2028-02-15 21:33:01.989280143 +0000 UTC" Jul 23 00:08:09.701012 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:09.700956 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13749h24m52.288328812s" Jul 23 00:08:09.941939 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:09.941903 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9853266_a585_469c_885e_79cc72f88d18.slice/crio-f7672af09ac6b9258013434fe3519e332cf50aabe3b814644b456e37aad2f03d WatchSource:0}: Error finding container f7672af09ac6b9258013434fe3519e332cf50aabe3b814644b456e37aad2f03d: Status 404 returned error can't find the container with id f7672af09ac6b9258013434fe3519e332cf50aabe3b814644b456e37aad2f03d Jul 23 00:08:09.945108 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:09.945059 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19f5db96_5df5_4b0c_8cc1_89d3f2719ed2.slice/crio-0ac794e62f2af5677ec09df7819f283eb5cb98178bed8a5216962f8762d03121 WatchSource:0}: Error finding container 0ac794e62f2af5677ec09df7819f283eb5cb98178bed8a5216962f8762d03121: Status 404 returned error can't find the container with id 0ac794e62f2af5677ec09df7819f283eb5cb98178bed8a5216962f8762d03121 Jul 23 00:08:09.947116 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:09.947088 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b166998_6f59_4909_b0aa_4c015ce61565.slice/crio-fc65451ded45cdc848e9bc0a64aceae3c18bfcb4baab312f11fd14598ffcbe57 WatchSource:0}: Error finding container fc65451ded45cdc848e9bc0a64aceae3c18bfcb4baab312f11fd14598ffcbe57: Status 404 returned error can't find the container with id fc65451ded45cdc848e9bc0a64aceae3c18bfcb4baab312f11fd14598ffcbe57 Jul 23 00:08:09.949017 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:09.948818 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d457c50_6bca_4609_8897_cef340b087e8.slice/crio-6bca4eeaff92e08e815010bdd90ab37a7d3d3d926a2c98dfabdc83a3ada4e9ae WatchSource:0}: Error finding container 6bca4eeaff92e08e815010bdd90ab37a7d3d3d926a2c98dfabdc83a3ada4e9ae: Status 404 returned error can't find the container with id 6bca4eeaff92e08e815010bdd90ab37a7d3d3d926a2c98dfabdc83a3ada4e9ae Jul 23 00:08:09.949193 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:09.949167 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod224cde93_d872_474d_b1c4_c9f92fc1f622.slice/crio-28ae9b0993cf3b10006f880854c49b253c535c330f6c90504a93901e5548e7b8 WatchSource:0}: Error finding container 28ae9b0993cf3b10006f880854c49b253c535c330f6c90504a93901e5548e7b8: Status 404 returned error can't find the container with id 28ae9b0993cf3b10006f880854c49b253c535c330f6c90504a93901e5548e7b8 Jul 23 00:08:09.951713 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:09.951551 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf5c6a95_1825_4855_9efb_906c85d29aa5.slice/crio-aed08c82d8b3ea0c342b61a24f57f35e97ad84837db8228aa2cdd6ab858650db WatchSource:0}: Error finding container aed08c82d8b3ea0c342b61a24f57f35e97ad84837db8228aa2cdd6ab858650db: Status 404 returned error can't find the container with id aed08c82d8b3ea0c342b61a24f57f35e97ad84837db8228aa2cdd6ab858650db Jul 23 00:08:09.952384 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:09.952361 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod216d84cd_f540_4bc0_9bd1_a8cc9e37367f.slice/crio-25704943eabff2718e296a65f05f9575e26c241a85caa37fac3047c8c1feb325 WatchSource:0}: Error finding container 25704943eabff2718e296a65f05f9575e26c241a85caa37fac3047c8c1feb325: Status 404 returned error can't find the container with id 25704943eabff2718e296a65f05f9575e26c241a85caa37fac3047c8c1feb325 Jul 23 00:08:09.953132 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:09.953103 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeaeb6df_4c50_464f_b5fa_a6e599d2fdf9.slice/crio-d5c6aa55fdf8824d099dbe11ba11228b81f4e1904c6ee4040bf6ab01d7d527d9 WatchSource:0}: Error finding container d5c6aa55fdf8824d099dbe11ba11228b81f4e1904c6ee4040bf6ab01d7d527d9: Status 404 returned error can't find the container with id d5c6aa55fdf8824d099dbe11ba11228b81f4e1904c6ee4040bf6ab01d7d527d9 Jul 23 00:08:09.954711 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:09.954656 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3749b807_a46e_43b6_880a_ac9bec9200f6.slice/crio-7f85fa53192372f7be61ce9669d909dd2a36e9f4a6e11eff5478d6af2439d26a WatchSource:0}: Error finding container 7f85fa53192372f7be61ce9669d909dd2a36e9f4a6e11eff5478d6af2439d26a: Status 404 returned error can't find the container with id 7f85fa53192372f7be61ce9669d909dd2a36e9f4a6e11eff5478d6af2439d26a Jul 23 00:08:10.000854 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:10.000827 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cgvgd" Jul 23 00:08:10.000972 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:10.000833 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jlsd6" Jul 23 00:08:10.001050 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:10.000832 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfgzz" Jul 23 00:08:10.001104 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:10.001058 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jlsd6" podUID="5f272157-a81d-439c-86ed-511e11146698" Jul 23 00:08:10.001157 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:10.001105 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfgzz" podUID="5a9bdee1-a493-4223-a7b6-3ae1f5024eb6" Jul 23 00:08:10.001157 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:10.000960 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cgvgd" podUID="72c0b00d-f59f-4186-b84e-470a1adac941" Jul 23 00:08:10.010495 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:10.010457 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jctrc" event={"ID":"aeaeb6df-4c50-464f-b5fa-a6e599d2fdf9","Type":"ContainerStarted","Data":"d5c6aa55fdf8824d099dbe11ba11228b81f4e1904c6ee4040bf6ab01d7d527d9"} Jul 23 00:08:10.011739 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:10.011708 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-sr9sh" event={"ID":"af5c6a95-1825-4855-9efb-906c85d29aa5","Type":"ContainerStarted","Data":"aed08c82d8b3ea0c342b61a24f57f35e97ad84837db8228aa2cdd6ab858650db"} Jul 23 00:08:10.012928 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:10.012905 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" event={"ID":"224cde93-d872-474d-b1c4-c9f92fc1f622","Type":"ContainerStarted","Data":"28ae9b0993cf3b10006f880854c49b253c535c330f6c90504a93901e5548e7b8"} Jul 23 00:08:10.014081 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:10.014054 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vgrs2" event={"ID":"9b166998-6f59-4909-b0aa-4c015ce61565","Type":"ContainerStarted","Data":"fc65451ded45cdc848e9bc0a64aceae3c18bfcb4baab312f11fd14598ffcbe57"} Jul 23 00:08:10.015160 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:10.015127 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvpdz" event={"ID":"3749b807-a46e-43b6-880a-ac9bec9200f6","Type":"ContainerStarted","Data":"7f85fa53192372f7be61ce9669d909dd2a36e9f4a6e11eff5478d6af2439d26a"} Jul 23 00:08:10.016201 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:10.016175 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8ghxf" event={"ID":"216d84cd-f540-4bc0-9bd1-a8cc9e37367f","Type":"ContainerStarted","Data":"25704943eabff2718e296a65f05f9575e26c241a85caa37fac3047c8c1feb325"} Jul 23 00:08:10.017270 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:10.017234 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9stwt" event={"ID":"7d457c50-6bca-4609-8897-cef340b087e8","Type":"ContainerStarted","Data":"6bca4eeaff92e08e815010bdd90ab37a7d3d3d926a2c98dfabdc83a3ada4e9ae"} Jul 23 00:08:10.018395 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:10.018367 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fhbjd" event={"ID":"19f5db96-5df5-4b0c-8cc1-89d3f2719ed2","Type":"ContainerStarted","Data":"0ac794e62f2af5677ec09df7819f283eb5cb98178bed8a5216962f8762d03121"} Jul 23 00:08:10.019568 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:10.019548 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-626nn" event={"ID":"f9853266-a585-469c-885e-79cc72f88d18","Type":"ContainerStarted","Data":"f7672af09ac6b9258013434fe3519e332cf50aabe3b814644b456e37aad2f03d"} Jul 23 00:08:10.290874 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:10.290562 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f272157-a81d-439c-86ed-511e11146698-original-pull-secret\") pod \"global-pull-secret-syncer-jlsd6\" (UID: \"5f272157-a81d-439c-86ed-511e11146698\") " pod="kube-system/global-pull-secret-syncer-jlsd6" Jul 23 00:08:10.290874 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:10.290827 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a9bdee1-a493-4223-a7b6-3ae1f5024eb6-metrics-certs\") pod \"network-metrics-daemon-kfgzz\" (UID: \"5a9bdee1-a493-4223-a7b6-3ae1f5024eb6\") " pod="openshift-multus/network-metrics-daemon-kfgzz" Jul 23 00:08:10.290874 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:10.290710 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 00:08:10.291076 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:10.290916 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 00:08:10.291076 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:10.290936 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f272157-a81d-439c-86ed-511e11146698-original-pull-secret podName:5f272157-a81d-439c-86ed-511e11146698 nodeName:}" failed. No retries permitted until 2026-07-23 00:08:12.29091716 +0000 UTC m=+6.086424012 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5f272157-a81d-439c-86ed-511e11146698-original-pull-secret") pod "global-pull-secret-syncer-jlsd6" (UID: "5f272157-a81d-439c-86ed-511e11146698") : object "kube-system"/"original-pull-secret" not registered Jul 23 00:08:10.291076 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:10.290956 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a9bdee1-a493-4223-a7b6-3ae1f5024eb6-metrics-certs podName:5a9bdee1-a493-4223-a7b6-3ae1f5024eb6 nodeName:}" failed. No retries permitted until 2026-07-23 00:08:12.290946727 +0000 UTC m=+6.086453566 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5a9bdee1-a493-4223-a7b6-3ae1f5024eb6-metrics-certs") pod "network-metrics-daemon-kfgzz" (UID: "5a9bdee1-a493-4223-a7b6-3ae1f5024eb6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 00:08:10.392499 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:10.391782 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2thdl\" (UniqueName: \"kubernetes.io/projected/72c0b00d-f59f-4186-b84e-470a1adac941-kube-api-access-2thdl\") pod \"network-check-target-cgvgd\" (UID: \"72c0b00d-f59f-4186-b84e-470a1adac941\") " pod="openshift-network-diagnostics/network-check-target-cgvgd" Jul 23 00:08:10.392499 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:10.392009 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 00:08:10.392499 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:10.392028 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 00:08:10.392499 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:10.392042 2562 projected.go:196] Error preparing data for projected volume kube-api-access-2thdl for pod openshift-network-diagnostics/network-check-target-cgvgd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 00:08:10.392499 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:10.392104 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/72c0b00d-f59f-4186-b84e-470a1adac941-kube-api-access-2thdl podName:72c0b00d-f59f-4186-b84e-470a1adac941 nodeName:}" failed. No retries permitted until 2026-07-23 00:08:12.392085705 +0000 UTC m=+6.187592543 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2thdl" (UniqueName: "kubernetes.io/projected/72c0b00d-f59f-4186-b84e-470a1adac941-kube-api-access-2thdl") pod "network-check-target-cgvgd" (UID: "72c0b00d-f59f-4186-b84e-470a1adac941") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 00:08:10.701686 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:10.701641 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 00:03:08 +0000 UTC" deadline="2028-04-26 05:50:35.62273522 +0000 UTC" Jul 23 00:08:10.701686 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:10.701684 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15437h42m24.921057142s" Jul 23 00:08:11.044614 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:11.043734 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-121.ec2.internal" event={"ID":"19e11d045e1035a525da0259bed898ce","Type":"ContainerStarted","Data":"4fb702e9e7924570128300be3e11fce2b29b9e770b027e8250a29a88b2fb9b9c"} Jul 23 00:08:11.053872 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:11.053231 2562 generic.go:356] "Generic (PLEG): container finished" podID="2dc6c03e0c457691dab513c68ebb04f1" containerID="2f1d01a3b135f31bc96e493ac9336b63d2e9382f67bed9cafa6bb5a315dd8c7f" exitCode=0 Jul 23 00:08:11.053872 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:11.053284 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-121.ec2.internal" event={"ID":"2dc6c03e0c457691dab513c68ebb04f1","Type":"ContainerDied","Data":"2f1d01a3b135f31bc96e493ac9336b63d2e9382f67bed9cafa6bb5a315dd8c7f"} Jul 23 00:08:11.062508 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:11.059268 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-121.ec2.internal" podStartSLOduration=4.059248008 podStartE2EDuration="4.059248008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:08:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:08:11.057927923 +0000 UTC m=+4.853434783" watchObservedRunningTime="2026-07-23 00:08:11.059248008 +0000 UTC m=+4.854754867" Jul 23 00:08:12.001676 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:12.001015 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jlsd6" Jul 23 00:08:12.001676 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:12.001189 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jlsd6" podUID="5f272157-a81d-439c-86ed-511e11146698" Jul 23 00:08:12.001676 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:12.001552 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cgvgd" Jul 23 00:08:12.001676 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:12.001662 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cgvgd" podUID="72c0b00d-f59f-4186-b84e-470a1adac941" Jul 23 00:08:12.002334 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:12.001727 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfgzz" Jul 23 00:08:12.002334 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:12.001857 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfgzz" podUID="5a9bdee1-a493-4223-a7b6-3ae1f5024eb6" Jul 23 00:08:12.066901 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:12.066816 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-121.ec2.internal" event={"ID":"2dc6c03e0c457691dab513c68ebb04f1","Type":"ContainerStarted","Data":"58c632bf79b94f991d74df2d48d74d51940deb85dd0fb6eb39e817b12791164e"} Jul 23 00:08:12.308888 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:12.308794 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f272157-a81d-439c-86ed-511e11146698-original-pull-secret\") pod \"global-pull-secret-syncer-jlsd6\" (UID: \"5f272157-a81d-439c-86ed-511e11146698\") " pod="kube-system/global-pull-secret-syncer-jlsd6" Jul 23 00:08:12.308888 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:12.308858 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a9bdee1-a493-4223-a7b6-3ae1f5024eb6-metrics-certs\") pod \"network-metrics-daemon-kfgzz\" (UID: \"5a9bdee1-a493-4223-a7b6-3ae1f5024eb6\") " pod="openshift-multus/network-metrics-daemon-kfgzz" Jul 23 00:08:12.309126 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:12.309083 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 00:08:12.309182 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:12.309151 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a9bdee1-a493-4223-a7b6-3ae1f5024eb6-metrics-certs podName:5a9bdee1-a493-4223-a7b6-3ae1f5024eb6 nodeName:}" failed. No retries permitted until 2026-07-23 00:08:16.309131396 +0000 UTC m=+10.104638277 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5a9bdee1-a493-4223-a7b6-3ae1f5024eb6-metrics-certs") pod "network-metrics-daemon-kfgzz" (UID: "5a9bdee1-a493-4223-a7b6-3ae1f5024eb6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 00:08:12.309663 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:12.309622 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 00:08:12.309748 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:12.309678 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f272157-a81d-439c-86ed-511e11146698-original-pull-secret podName:5f272157-a81d-439c-86ed-511e11146698 nodeName:}" failed. No retries permitted until 2026-07-23 00:08:16.309662479 +0000 UTC m=+10.105169319 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5f272157-a81d-439c-86ed-511e11146698-original-pull-secret") pod "global-pull-secret-syncer-jlsd6" (UID: "5f272157-a81d-439c-86ed-511e11146698") : object "kube-system"/"original-pull-secret" not registered Jul 23 00:08:12.409491 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:12.409443 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2thdl\" (UniqueName: \"kubernetes.io/projected/72c0b00d-f59f-4186-b84e-470a1adac941-kube-api-access-2thdl\") pod \"network-check-target-cgvgd\" (UID: \"72c0b00d-f59f-4186-b84e-470a1adac941\") " pod="openshift-network-diagnostics/network-check-target-cgvgd" Jul 23 00:08:12.409846 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:12.409795 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 00:08:12.409846 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:12.409826 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 00:08:12.409846 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:12.409840 2562 projected.go:196] Error preparing data for projected volume kube-api-access-2thdl for pod openshift-network-diagnostics/network-check-target-cgvgd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 00:08:12.410060 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:12.409904 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/72c0b00d-f59f-4186-b84e-470a1adac941-kube-api-access-2thdl podName:72c0b00d-f59f-4186-b84e-470a1adac941 nodeName:}" failed. No retries permitted until 2026-07-23 00:08:16.409885252 +0000 UTC m=+10.205392089 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2thdl" (UniqueName: "kubernetes.io/projected/72c0b00d-f59f-4186-b84e-470a1adac941-kube-api-access-2thdl") pod "network-check-target-cgvgd" (UID: "72c0b00d-f59f-4186-b84e-470a1adac941") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 00:08:14.001369 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:14.001096 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfgzz" Jul 23 00:08:14.001369 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:14.001249 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfgzz" podUID="5a9bdee1-a493-4223-a7b6-3ae1f5024eb6" Jul 23 00:08:14.001921 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:14.001631 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cgvgd" Jul 23 00:08:14.001921 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:14.001749 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cgvgd" podUID="72c0b00d-f59f-4186-b84e-470a1adac941" Jul 23 00:08:14.001921 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:14.001774 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jlsd6" Jul 23 00:08:14.001921 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:14.001870 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jlsd6" podUID="5f272157-a81d-439c-86ed-511e11146698" Jul 23 00:08:16.000930 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:16.000890 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jlsd6" Jul 23 00:08:16.001409 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:16.001035 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jlsd6" podUID="5f272157-a81d-439c-86ed-511e11146698" Jul 23 00:08:16.001409 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:16.001380 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cgvgd" Jul 23 00:08:16.001537 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:16.001519 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cgvgd" podUID="72c0b00d-f59f-4186-b84e-470a1adac941" Jul 23 00:08:16.001597 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:16.001555 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfgzz" Jul 23 00:08:16.001668 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:16.001650 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfgzz" podUID="5a9bdee1-a493-4223-a7b6-3ae1f5024eb6" Jul 23 00:08:16.344167 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:16.344075 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f272157-a81d-439c-86ed-511e11146698-original-pull-secret\") pod \"global-pull-secret-syncer-jlsd6\" (UID: \"5f272157-a81d-439c-86ed-511e11146698\") " pod="kube-system/global-pull-secret-syncer-jlsd6" Jul 23 00:08:16.344328 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:16.344214 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a9bdee1-a493-4223-a7b6-3ae1f5024eb6-metrics-certs\") pod \"network-metrics-daemon-kfgzz\" (UID: \"5a9bdee1-a493-4223-a7b6-3ae1f5024eb6\") " pod="openshift-multus/network-metrics-daemon-kfgzz" Jul 23 00:08:16.344386 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:16.344367 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 00:08:16.344459 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:16.344431 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 00:08:16.344459 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:16.344450 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f272157-a81d-439c-86ed-511e11146698-original-pull-secret podName:5f272157-a81d-439c-86ed-511e11146698 nodeName:}" failed. No retries permitted until 2026-07-23 00:08:24.344428932 +0000 UTC m=+18.139935789 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5f272157-a81d-439c-86ed-511e11146698-original-pull-secret") pod "global-pull-secret-syncer-jlsd6" (UID: "5f272157-a81d-439c-86ed-511e11146698") : object "kube-system"/"original-pull-secret" not registered Jul 23 00:08:16.344663 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:16.344491 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a9bdee1-a493-4223-a7b6-3ae1f5024eb6-metrics-certs podName:5a9bdee1-a493-4223-a7b6-3ae1f5024eb6 nodeName:}" failed. No retries permitted until 2026-07-23 00:08:24.344478373 +0000 UTC m=+18.139985226 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5a9bdee1-a493-4223-a7b6-3ae1f5024eb6-metrics-certs") pod "network-metrics-daemon-kfgzz" (UID: "5a9bdee1-a493-4223-a7b6-3ae1f5024eb6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 00:08:16.444789 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:16.444730 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2thdl\" (UniqueName: \"kubernetes.io/projected/72c0b00d-f59f-4186-b84e-470a1adac941-kube-api-access-2thdl\") pod \"network-check-target-cgvgd\" (UID: \"72c0b00d-f59f-4186-b84e-470a1adac941\") " pod="openshift-network-diagnostics/network-check-target-cgvgd" Jul 23 00:08:16.444985 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:16.444959 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 00:08:16.445048 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:16.444986 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 00:08:16.445048 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:16.445001 2562 projected.go:196] Error preparing data for projected volume kube-api-access-2thdl for pod openshift-network-diagnostics/network-check-target-cgvgd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 00:08:16.445154 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:16.445088 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/72c0b00d-f59f-4186-b84e-470a1adac941-kube-api-access-2thdl podName:72c0b00d-f59f-4186-b84e-470a1adac941 nodeName:}" failed. No retries permitted until 2026-07-23 00:08:24.445067688 +0000 UTC m=+18.240574546 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2thdl" (UniqueName: "kubernetes.io/projected/72c0b00d-f59f-4186-b84e-470a1adac941-kube-api-access-2thdl") pod "network-check-target-cgvgd" (UID: "72c0b00d-f59f-4186-b84e-470a1adac941") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 00:08:18.001462 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:18.001427 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jlsd6" Jul 23 00:08:18.002047 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:18.001473 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfgzz" Jul 23 00:08:18.002047 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:18.001477 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cgvgd" Jul 23 00:08:18.002047 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:18.001564 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jlsd6" podUID="5f272157-a81d-439c-86ed-511e11146698" Jul 23 00:08:18.002047 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:18.001686 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cgvgd" podUID="72c0b00d-f59f-4186-b84e-470a1adac941" Jul 23 00:08:18.002047 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:18.001842 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfgzz" podUID="5a9bdee1-a493-4223-a7b6-3ae1f5024eb6" Jul 23 00:08:20.000628 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:20.000582 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cgvgd" Jul 23 00:08:20.000628 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:20.000607 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfgzz" Jul 23 00:08:20.001120 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:20.000677 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jlsd6" Jul 23 00:08:20.001120 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:20.000691 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cgvgd" podUID="72c0b00d-f59f-4186-b84e-470a1adac941" Jul 23 00:08:20.001120 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:20.000778 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jlsd6" podUID="5f272157-a81d-439c-86ed-511e11146698" Jul 23 00:08:20.001120 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:20.000862 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfgzz" podUID="5a9bdee1-a493-4223-a7b6-3ae1f5024eb6" Jul 23 00:08:22.001219 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:22.001176 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cgvgd" Jul 23 00:08:22.001219 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:22.001203 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jlsd6" Jul 23 00:08:22.001702 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:22.001176 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfgzz" Jul 23 00:08:22.001702 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:22.001304 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cgvgd" podUID="72c0b00d-f59f-4186-b84e-470a1adac941" Jul 23 00:08:22.001702 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:22.001408 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfgzz" podUID="5a9bdee1-a493-4223-a7b6-3ae1f5024eb6" Jul 23 00:08:22.001702 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:22.001495 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jlsd6" podUID="5f272157-a81d-439c-86ed-511e11146698" Jul 23 00:08:24.001364 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:24.001325 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfgzz" Jul 23 00:08:24.001900 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:24.001326 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jlsd6" Jul 23 00:08:24.001900 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:24.001467 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfgzz" podUID="5a9bdee1-a493-4223-a7b6-3ae1f5024eb6" Jul 23 00:08:24.001900 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:24.001325 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cgvgd" Jul 23 00:08:24.001900 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:24.001519 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jlsd6" podUID="5f272157-a81d-439c-86ed-511e11146698" Jul 23 00:08:24.001900 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:24.001625 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cgvgd" podUID="72c0b00d-f59f-4186-b84e-470a1adac941" Jul 23 00:08:24.407198 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:24.407161 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f272157-a81d-439c-86ed-511e11146698-original-pull-secret\") pod \"global-pull-secret-syncer-jlsd6\" (UID: \"5f272157-a81d-439c-86ed-511e11146698\") " pod="kube-system/global-pull-secret-syncer-jlsd6" Jul 23 00:08:24.407198 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:24.407208 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a9bdee1-a493-4223-a7b6-3ae1f5024eb6-metrics-certs\") pod \"network-metrics-daemon-kfgzz\" (UID: \"5a9bdee1-a493-4223-a7b6-3ae1f5024eb6\") " pod="openshift-multus/network-metrics-daemon-kfgzz" Jul 23 00:08:24.407422 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:24.407316 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 00:08:24.407422 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:24.407338 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 00:08:24.407422 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:24.407382 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f272157-a81d-439c-86ed-511e11146698-original-pull-secret podName:5f272157-a81d-439c-86ed-511e11146698 nodeName:}" failed. No retries permitted until 2026-07-23 00:08:40.407367373 +0000 UTC m=+34.202874208 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5f272157-a81d-439c-86ed-511e11146698-original-pull-secret") pod "global-pull-secret-syncer-jlsd6" (UID: "5f272157-a81d-439c-86ed-511e11146698") : object "kube-system"/"original-pull-secret" not registered Jul 23 00:08:24.407422 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:24.407398 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a9bdee1-a493-4223-a7b6-3ae1f5024eb6-metrics-certs podName:5a9bdee1-a493-4223-a7b6-3ae1f5024eb6 nodeName:}" failed. No retries permitted until 2026-07-23 00:08:40.407390777 +0000 UTC m=+34.202897613 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5a9bdee1-a493-4223-a7b6-3ae1f5024eb6-metrics-certs") pod "network-metrics-daemon-kfgzz" (UID: "5a9bdee1-a493-4223-a7b6-3ae1f5024eb6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 00:08:24.508200 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:24.508155 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2thdl\" (UniqueName: \"kubernetes.io/projected/72c0b00d-f59f-4186-b84e-470a1adac941-kube-api-access-2thdl\") pod \"network-check-target-cgvgd\" (UID: \"72c0b00d-f59f-4186-b84e-470a1adac941\") " pod="openshift-network-diagnostics/network-check-target-cgvgd" Jul 23 00:08:24.508350 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:24.508284 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 00:08:24.508350 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:24.508299 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 00:08:24.508350 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:24.508308 2562 projected.go:196] Error preparing data for projected volume kube-api-access-2thdl for pod openshift-network-diagnostics/network-check-target-cgvgd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 00:08:24.508468 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:24.508359 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/72c0b00d-f59f-4186-b84e-470a1adac941-kube-api-access-2thdl podName:72c0b00d-f59f-4186-b84e-470a1adac941 nodeName:}" failed. No retries permitted until 2026-07-23 00:08:40.508346252 +0000 UTC m=+34.303853086 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2thdl" (UniqueName: "kubernetes.io/projected/72c0b00d-f59f-4186-b84e-470a1adac941-kube-api-access-2thdl") pod "network-check-target-cgvgd" (UID: "72c0b00d-f59f-4186-b84e-470a1adac941") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 00:08:26.001539 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:26.001496 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cgvgd" Jul 23 00:08:26.001960 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:26.001629 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfgzz" Jul 23 00:08:26.001960 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:26.001637 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cgvgd" podUID="72c0b00d-f59f-4186-b84e-470a1adac941" Jul 23 00:08:26.001960 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:26.001670 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jlsd6" Jul 23 00:08:26.001960 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:26.001795 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfgzz" podUID="5a9bdee1-a493-4223-a7b6-3ae1f5024eb6" Jul 23 00:08:26.001960 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:26.001884 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jlsd6" podUID="5f272157-a81d-439c-86ed-511e11146698" Jul 23 00:08:28.000643 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:28.000602 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jlsd6" Jul 23 00:08:28.001106 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:28.000602 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cgvgd" Jul 23 00:08:28.001106 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:28.000718 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jlsd6" podUID="5f272157-a81d-439c-86ed-511e11146698" Jul 23 00:08:28.001106 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:28.000847 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cgvgd" podUID="72c0b00d-f59f-4186-b84e-470a1adac941" Jul 23 00:08:28.001106 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:28.000606 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfgzz" Jul 23 00:08:28.001106 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:28.000965 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfgzz" podUID="5a9bdee1-a493-4223-a7b6-3ae1f5024eb6" Jul 23 00:08:29.103313 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:29.102900 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvpdz" event={"ID":"3749b807-a46e-43b6-880a-ac9bec9200f6","Type":"ContainerStarted","Data":"b061c97e696aa5ed2878fdee226a0c763cb91f5e987dfa8d71f3a1f4efe5f9ab"} Jul 23 00:08:29.104678 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:29.104653 2562 generic.go:356] "Generic (PLEG): container finished" podID="216d84cd-f540-4bc0-9bd1-a8cc9e37367f" containerID="e619c9bf5f2a2885b3873f59ebbcf58c80b3458d762381a73189e700cfda4245" exitCode=0 Jul 23 00:08:29.104807 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:29.104725 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8ghxf" event={"ID":"216d84cd-f540-4bc0-9bd1-a8cc9e37367f","Type":"ContainerDied","Data":"e619c9bf5f2a2885b3873f59ebbcf58c80b3458d762381a73189e700cfda4245"} Jul 23 00:08:29.106362 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:29.106336 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9stwt" event={"ID":"7d457c50-6bca-4609-8897-cef340b087e8","Type":"ContainerStarted","Data":"ad96ea9108637ec3691ef0c147890846d03da8c62cc384ba583da25f025859d1"} Jul 23 00:08:29.107794 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:29.107771 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fhbjd" event={"ID":"19f5db96-5df5-4b0c-8cc1-89d3f2719ed2","Type":"ContainerStarted","Data":"0312a015e2152032c5eab5e175c30113117446a2c29549c09e016340d547ebc7"} Jul 23 00:08:29.109231 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:29.109202 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jctrc" event={"ID":"aeaeb6df-4c50-464f-b5fa-a6e599d2fdf9","Type":"ContainerStarted","Data":"a4247010db4de53c1d1dccc82cfae9998fd9ea9e96ae7d6bdc86234e5d53e7e7"} Jul 23 00:08:29.110712 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:29.110685 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-sr9sh" event={"ID":"af5c6a95-1825-4855-9efb-906c85d29aa5","Type":"ContainerStarted","Data":"267e1629e9b656a7ece8e801a9854385dd0bbe0c0aecef1c38ae6084d25fe8a1"} Jul 23 00:08:29.112662 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:29.112636 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" event={"ID":"224cde93-d872-474d-b1c4-c9f92fc1f622","Type":"ContainerStarted","Data":"3984da12edb06f7c9633da4384c781b74bdb6d3ace6c7a8be4ca2c1542d184c6"} Jul 23 00:08:29.112770 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:29.112712 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" event={"ID":"224cde93-d872-474d-b1c4-c9f92fc1f622","Type":"ContainerStarted","Data":"669ba0e0858793dd390e00330cbc2cc9b66ca5b4ff81dbc233121d69dca0c298"} Jul 23 00:08:29.112826 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:29.112784 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" event={"ID":"224cde93-d872-474d-b1c4-c9f92fc1f622","Type":"ContainerStarted","Data":"4d417c988b465360091ccfac93fb41b97213ec40ad30339a205db16e1077ba46"} Jul 23 00:08:29.114523 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:29.114504 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vgrs2" event={"ID":"9b166998-6f59-4909-b0aa-4c015ce61565","Type":"ContainerStarted","Data":"5b122ce05ab738f52b0ce06cdcd23d46460edd69f0c89f1ac99f86f5392ddadf"} Jul 23 00:08:29.126979 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:29.126939 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-121.ec2.internal" podStartSLOduration=22.12692449 podStartE2EDuration="22.12692449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:08:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:08:12.082468626 +0000 UTC m=+5.877975487" watchObservedRunningTime="2026-07-23 00:08:29.12692449 +0000 UTC m=+22.922431346" Jul 23 00:08:29.142494 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:29.142444 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-9stwt" podStartSLOduration=3.402894049 podStartE2EDuration="22.142429306s" totalImagesPullingTime="18.739535257s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:08:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:08:29.142084149 +0000 UTC m=+22.937591007" watchObservedRunningTime="2026-07-23 00:08:29.142429306 +0000 UTC m=+22.937936162" Jul 23 00:08:29.156252 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:29.156205 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jctrc" podStartSLOduration=3.422205071 podStartE2EDuration="22.156189852s" totalImagesPullingTime="18.733984781s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:08:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:08:29.155770718 +0000 UTC m=+22.951277572" watchObservedRunningTime="2026-07-23 00:08:29.156189852 +0000 UTC m=+22.951696710" Jul 23 00:08:29.187205 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:29.187152 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-sr9sh" podStartSLOduration=4.447399106 podStartE2EDuration="23.187135795s" totalImagesPullingTime="18.739736689s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:08:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:08:29.172001187 +0000 UTC m=+22.967508045" watchObservedRunningTime="2026-07-23 00:08:29.187135795 +0000 UTC m=+22.982642630" Jul 23 00:08:29.187522 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:29.187497 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-fhbjd" podStartSLOduration=11.542272797 podStartE2EDuration="22.18749071s" totalImagesPullingTime="10.645217913s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:08:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:08:29.187049473 +0000 UTC m=+22.982556331" watchObservedRunningTime="2026-07-23 00:08:29.18749071 +0000 UTC m=+22.982997566" Jul 23 00:08:29.205805 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:29.205739 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-vgrs2" podStartSLOduration=3.410040484 podStartE2EDuration="22.205724034s" totalImagesPullingTime="18.79568355s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:08:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:08:29.205460396 +0000 UTC m=+23.000967286" watchObservedRunningTime="2026-07-23 00:08:29.205724034 +0000 UTC m=+23.001230887" Jul 23 00:08:30.000929 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:30.000888 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jlsd6" Jul 23 00:08:30.001203 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:30.000907 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfgzz" Jul 23 00:08:30.001203 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:30.001020 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jlsd6" podUID="5f272157-a81d-439c-86ed-511e11146698" Jul 23 00:08:30.001203 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:30.001122 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfgzz" podUID="5a9bdee1-a493-4223-a7b6-3ae1f5024eb6" Jul 23 00:08:30.001203 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:30.000888 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cgvgd" Jul 23 00:08:30.001389 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:30.001223 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cgvgd" podUID="72c0b00d-f59f-4186-b84e-470a1adac941" Jul 23 00:08:30.121154 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:30.121110 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" event={"ID":"224cde93-d872-474d-b1c4-c9f92fc1f622","Type":"ContainerStarted","Data":"e5d9afd55e421baba9bbbf5444d1ec13f45566ba7114b1f29957b4b970feebf2"} Jul 23 00:08:30.121588 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:30.121205 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" event={"ID":"224cde93-d872-474d-b1c4-c9f92fc1f622","Type":"ContainerStarted","Data":"c738c908b3bb8f955610f82a6c5c91e60f0247286b0a3f225e7d1d96983986fd"} Jul 23 00:08:30.121588 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:30.121252 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" event={"ID":"224cde93-d872-474d-b1c4-c9f92fc1f622","Type":"ContainerStarted","Data":"41b46a5d71d0fed1772c587ec6e8dc2f4f5c04462284a49cb398ddb7d0e79d7b"} Jul 23 00:08:30.122907 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:30.122873 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-626nn" event={"ID":"f9853266-a585-469c-885e-79cc72f88d18","Type":"ContainerStarted","Data":"9d2c7e9e72969c3170b809d9b635ed861eae11b854971ff5cb824ba30c72a8d8"} Jul 23 00:08:30.138628 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:30.138565 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-626nn" podStartSLOduration=4.391605878 podStartE2EDuration="23.138546314s" totalImagesPullingTime="18.746940436s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:08:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:08:30.137487656 +0000 UTC m=+23.932994515" watchObservedRunningTime="2026-07-23 00:08:30.138546314 +0000 UTC m=+23.934053173" Jul 23 00:08:30.367599 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:30.367568 2562 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 23 00:08:30.879919 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:30.879792 2562 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-23T00:08:30.367591187Z","UUID":"a0e09cce-93aa-4cf9-9278-2a7edc47c05d","Handler":null,"Name":"","Endpoint":""} Jul 23 00:08:30.881508 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:30.881477 2562 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 23 00:08:30.881680 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:30.881522 2562 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 23 00:08:31.127534 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:31.127501 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvpdz" event={"ID":"3749b807-a46e-43b6-880a-ac9bec9200f6","Type":"ContainerStarted","Data":"b4dc305ccdd240c25606ff1395642f18292ae69d4a23e3baf3a8bd922e8d6e10"} Jul 23 00:08:32.001203 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:32.000922 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cgvgd" Jul 23 00:08:32.001432 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:32.000949 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfgzz" Jul 23 00:08:32.001432 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:32.001251 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cgvgd" podUID="72c0b00d-f59f-4186-b84e-470a1adac941" Jul 23 00:08:32.001432 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:32.000951 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jlsd6" Jul 23 00:08:32.001432 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:32.001362 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfgzz" podUID="5a9bdee1-a493-4223-a7b6-3ae1f5024eb6" Jul 23 00:08:32.001627 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:32.001459 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jlsd6" podUID="5f272157-a81d-439c-86ed-511e11146698" Jul 23 00:08:32.136377 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:32.136339 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvpdz" event={"ID":"3749b807-a46e-43b6-880a-ac9bec9200f6","Type":"ContainerStarted","Data":"731a01116fb58d5744bb67ee51a2a3c50c405a98f1a1916b8c8f04c2fe7eb835"} Jul 23 00:08:32.140529 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:32.140486 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" event={"ID":"224cde93-d872-474d-b1c4-c9f92fc1f622","Type":"ContainerStarted","Data":"ee5ebf3fdc6718910fa265b3b19e6982db4b9130005c5e2f8ff389ecc20bcf8d"} Jul 23 00:08:32.685052 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:32.685009 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-fhbjd" Jul 23 00:08:32.685683 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:32.685660 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-fhbjd" Jul 23 00:08:32.699921 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:32.699864 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mvpdz" podStartSLOduration=4.432783775 podStartE2EDuration="25.699842942s" totalImagesPullingTime="21.267059167s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:08:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:08:32.169705255 +0000 UTC m=+25.965212114" watchObservedRunningTime="2026-07-23 00:08:32.699842942 +0000 UTC m=+26.495349802" Jul 23 00:08:33.143034 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:33.142979 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-fhbjd" Jul 23 00:08:33.143542 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:33.143317 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-fhbjd" Jul 23 00:08:34.001663 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:34.001467 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jlsd6" Jul 23 00:08:34.001835 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:34.001490 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfgzz" Jul 23 00:08:34.001835 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:34.001739 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jlsd6" podUID="5f272157-a81d-439c-86ed-511e11146698" Jul 23 00:08:34.001902 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:34.001849 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfgzz" podUID="5a9bdee1-a493-4223-a7b6-3ae1f5024eb6" Jul 23 00:08:34.001902 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:34.001516 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cgvgd" Jul 23 00:08:34.001972 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:34.001929 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cgvgd" podUID="72c0b00d-f59f-4186-b84e-470a1adac941" Jul 23 00:08:34.146583 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:34.146472 2562 generic.go:356] "Generic (PLEG): container finished" podID="216d84cd-f540-4bc0-9bd1-a8cc9e37367f" containerID="c9dd9d8b7d976352393839dba1d1f89679d93eb1b7ee79e83727e5e495a2967e" exitCode=0 Jul 23 00:08:34.146583 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:34.146557 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8ghxf" event={"ID":"216d84cd-f540-4bc0-9bd1-a8cc9e37367f","Type":"ContainerDied","Data":"c9dd9d8b7d976352393839dba1d1f89679d93eb1b7ee79e83727e5e495a2967e"} Jul 23 00:08:34.150225 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:34.150188 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" event={"ID":"224cde93-d872-474d-b1c4-c9f92fc1f622","Type":"ContainerStarted","Data":"49f50164625ffc1e12349fc386a6b966c9e9d4fef35f91fb695746b4cef061d4"} Jul 23 00:08:34.150576 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:34.150515 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" Jul 23 00:08:34.150679 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:34.150599 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" Jul 23 00:08:34.150679 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:34.150643 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" Jul 23 00:08:34.167370 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:34.167342 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" Jul 23 00:08:34.167522 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:34.167482 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" Jul 23 00:08:34.197510 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:34.197439 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" podStartSLOduration=8.386225886 podStartE2EDuration="27.197418742s" totalImagesPullingTime="18.811192856s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:08:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:08:34.192313359 +0000 UTC m=+27.987820217" watchObservedRunningTime="2026-07-23 00:08:34.197418742 +0000 UTC m=+27.992925601" Jul 23 00:08:35.755007 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:35.754971 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kfgzz"] Jul 23 00:08:35.755492 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:35.755138 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfgzz" Jul 23 00:08:35.755492 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:35.755247 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfgzz" podUID="5a9bdee1-a493-4223-a7b6-3ae1f5024eb6" Jul 23 00:08:35.758220 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:35.758193 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jlsd6"] Jul 23 00:08:35.758363 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:35.758308 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jlsd6" Jul 23 00:08:35.758435 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:35.758410 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jlsd6" podUID="5f272157-a81d-439c-86ed-511e11146698" Jul 23 00:08:35.758829 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:35.758803 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cgvgd"] Jul 23 00:08:35.758944 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:35.758907 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cgvgd" Jul 23 00:08:35.759018 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:35.758998 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cgvgd" podUID="72c0b00d-f59f-4186-b84e-470a1adac941" Jul 23 00:08:36.156823 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:36.156786 2562 generic.go:356] "Generic (PLEG): container finished" podID="216d84cd-f540-4bc0-9bd1-a8cc9e37367f" containerID="337fed2344f6ab3c77e8a61d76a8ddd1c595804a967719bb7dc96a6a1bc24368" exitCode=0 Jul 23 00:08:36.156997 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:36.156873 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8ghxf" event={"ID":"216d84cd-f540-4bc0-9bd1-a8cc9e37367f","Type":"ContainerDied","Data":"337fed2344f6ab3c77e8a61d76a8ddd1c595804a967719bb7dc96a6a1bc24368"} Jul 23 00:08:37.001676 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:37.001637 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cgvgd" Jul 23 00:08:37.002071 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:37.001730 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cgvgd" podUID="72c0b00d-f59f-4186-b84e-470a1adac941" Jul 23 00:08:38.000666 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:38.000628 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfgzz" Jul 23 00:08:38.000666 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:38.000658 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jlsd6" Jul 23 00:08:38.000904 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:38.000780 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jlsd6" podUID="5f272157-a81d-439c-86ed-511e11146698" Jul 23 00:08:38.000947 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:38.000903 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfgzz" podUID="5a9bdee1-a493-4223-a7b6-3ae1f5024eb6" Jul 23 00:08:38.165216 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:38.165145 2562 generic.go:356] "Generic (PLEG): container finished" podID="216d84cd-f540-4bc0-9bd1-a8cc9e37367f" containerID="e046ce1e81e705de23a3e210f22660e0401c41d005e8b5b1930f02ef16d4fcdb" exitCode=0 Jul 23 00:08:38.165216 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:38.165191 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8ghxf" event={"ID":"216d84cd-f540-4bc0-9bd1-a8cc9e37367f","Type":"ContainerDied","Data":"e046ce1e81e705de23a3e210f22660e0401c41d005e8b5b1930f02ef16d4fcdb"} Jul 23 00:08:39.001279 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:39.001246 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cgvgd" Jul 23 00:08:39.001490 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:39.001387 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cgvgd" podUID="72c0b00d-f59f-4186-b84e-470a1adac941" Jul 23 00:08:40.000926 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:40.000888 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jlsd6" Jul 23 00:08:40.001630 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:40.000897 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfgzz" Jul 23 00:08:40.001630 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:40.001017 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jlsd6" podUID="5f272157-a81d-439c-86ed-511e11146698" Jul 23 00:08:40.001630 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:40.001145 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfgzz" podUID="5a9bdee1-a493-4223-a7b6-3ae1f5024eb6" Jul 23 00:08:40.427345 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:40.427307 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f272157-a81d-439c-86ed-511e11146698-original-pull-secret\") pod \"global-pull-secret-syncer-jlsd6\" (UID: \"5f272157-a81d-439c-86ed-511e11146698\") " pod="kube-system/global-pull-secret-syncer-jlsd6" Jul 23 00:08:40.427345 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:40.427357 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a9bdee1-a493-4223-a7b6-3ae1f5024eb6-metrics-certs\") pod \"network-metrics-daemon-kfgzz\" (UID: \"5a9bdee1-a493-4223-a7b6-3ae1f5024eb6\") " pod="openshift-multus/network-metrics-daemon-kfgzz" Jul 23 00:08:40.427593 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:40.427484 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 00:08:40.427593 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:40.427486 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 00:08:40.427593 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:40.427553 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a9bdee1-a493-4223-a7b6-3ae1f5024eb6-metrics-certs podName:5a9bdee1-a493-4223-a7b6-3ae1f5024eb6 nodeName:}" failed. No retries permitted until 2026-07-23 00:09:12.427534359 +0000 UTC m=+66.223041235 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5a9bdee1-a493-4223-a7b6-3ae1f5024eb6-metrics-certs") pod "network-metrics-daemon-kfgzz" (UID: "5a9bdee1-a493-4223-a7b6-3ae1f5024eb6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 00:08:40.427593 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:40.427570 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f272157-a81d-439c-86ed-511e11146698-original-pull-secret podName:5f272157-a81d-439c-86ed-511e11146698 nodeName:}" failed. No retries permitted until 2026-07-23 00:09:12.42756327 +0000 UTC m=+66.223070105 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5f272157-a81d-439c-86ed-511e11146698-original-pull-secret") pod "global-pull-secret-syncer-jlsd6" (UID: "5f272157-a81d-439c-86ed-511e11146698") : object "kube-system"/"original-pull-secret" not registered Jul 23 00:08:40.528130 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:40.528080 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2thdl\" (UniqueName: \"kubernetes.io/projected/72c0b00d-f59f-4186-b84e-470a1adac941-kube-api-access-2thdl\") pod \"network-check-target-cgvgd\" (UID: \"72c0b00d-f59f-4186-b84e-470a1adac941\") " pod="openshift-network-diagnostics/network-check-target-cgvgd" Jul 23 00:08:40.528315 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:40.528251 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 00:08:40.528315 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:40.528278 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 00:08:40.528315 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:40.528289 2562 projected.go:196] Error preparing data for projected volume kube-api-access-2thdl for pod openshift-network-diagnostics/network-check-target-cgvgd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 00:08:40.528461 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:40.528344 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/72c0b00d-f59f-4186-b84e-470a1adac941-kube-api-access-2thdl podName:72c0b00d-f59f-4186-b84e-470a1adac941 nodeName:}" failed. No retries permitted until 2026-07-23 00:09:12.528330199 +0000 UTC m=+66.323837037 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-2thdl" (UniqueName: "kubernetes.io/projected/72c0b00d-f59f-4186-b84e-470a1adac941-kube-api-access-2thdl") pod "network-check-target-cgvgd" (UID: "72c0b00d-f59f-4186-b84e-470a1adac941") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 00:08:41.001774 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:41.001721 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cgvgd" Jul 23 00:08:41.002388 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:41.001870 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cgvgd" podUID="72c0b00d-f59f-4186-b84e-470a1adac941" Jul 23 00:08:41.504430 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:41.504394 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-121.ec2.internal" event="NodeReady" Jul 23 00:08:41.504618 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:41.504582 2562 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 23 00:08:41.585621 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:41.585583 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-ndkl6"] Jul 23 00:08:41.611493 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:41.611458 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-pqp48"] Jul 23 00:08:41.611673 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:41.611608 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ndkl6" Jul 23 00:08:41.614403 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:41.614372 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 23 00:08:41.614564 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:41.614446 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fhhqg\"" Jul 23 00:08:41.614564 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:41.614446 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 23 00:08:41.631478 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:41.631449 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pqp48"] Jul 23 00:08:41.631478 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:41.631486 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ndkl6"] Jul 23 00:08:41.631715 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:41.631571 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pqp48" Jul 23 00:08:41.634253 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:41.634224 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 23 00:08:41.634253 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:41.634246 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 23 00:08:41.634430 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:41.634227 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-httr9\"" Jul 23 00:08:41.634430 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:41.634223 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 23 00:08:41.736997 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:41.736962 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd67adef-fce4-429b-b01b-8225f9c33ab4-tmp-dir\") pod \"dns-default-ndkl6\" (UID: \"dd67adef-fce4-429b-b01b-8225f9c33ab4\") " pod="openshift-dns/dns-default-ndkl6" Jul 23 00:08:41.737190 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:41.737084 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dd67adef-fce4-429b-b01b-8225f9c33ab4-config-volume\") pod \"dns-default-ndkl6\" (UID: \"dd67adef-fce4-429b-b01b-8225f9c33ab4\") " pod="openshift-dns/dns-default-ndkl6" Jul 23 00:08:41.737190 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:41.737116 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dd67adef-fce4-429b-b01b-8225f9c33ab4-metrics-tls\") pod \"dns-default-ndkl6\" (UID: \"dd67adef-fce4-429b-b01b-8225f9c33ab4\") " pod="openshift-dns/dns-default-ndkl6" Jul 23 00:08:41.737190 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:41.737164 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c1bab695-9ea2-4498-bf9e-e750b5850fcc-cert\") pod \"ingress-canary-pqp48\" (UID: \"c1bab695-9ea2-4498-bf9e-e750b5850fcc\") " pod="openshift-ingress-canary/ingress-canary-pqp48" Jul 23 00:08:41.737190 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:41.737188 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx5gg\" (UniqueName: \"kubernetes.io/projected/dd67adef-fce4-429b-b01b-8225f9c33ab4-kube-api-access-gx5gg\") pod \"dns-default-ndkl6\" (UID: \"dd67adef-fce4-429b-b01b-8225f9c33ab4\") " pod="openshift-dns/dns-default-ndkl6" Jul 23 00:08:41.737406 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:41.737272 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq72q\" (UniqueName: \"kubernetes.io/projected/c1bab695-9ea2-4498-bf9e-e750b5850fcc-kube-api-access-bq72q\") pod \"ingress-canary-pqp48\" (UID: \"c1bab695-9ea2-4498-bf9e-e750b5850fcc\") " pod="openshift-ingress-canary/ingress-canary-pqp48" Jul 23 00:08:41.838391 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:41.838344 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dd67adef-fce4-429b-b01b-8225f9c33ab4-config-volume\") pod \"dns-default-ndkl6\" (UID: \"dd67adef-fce4-429b-b01b-8225f9c33ab4\") " pod="openshift-dns/dns-default-ndkl6" Jul 23 00:08:41.838581 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:41.838404 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dd67adef-fce4-429b-b01b-8225f9c33ab4-metrics-tls\") pod \"dns-default-ndkl6\" (UID: \"dd67adef-fce4-429b-b01b-8225f9c33ab4\") " pod="openshift-dns/dns-default-ndkl6" Jul 23 00:08:41.838581 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:41.838446 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c1bab695-9ea2-4498-bf9e-e750b5850fcc-cert\") pod \"ingress-canary-pqp48\" (UID: \"c1bab695-9ea2-4498-bf9e-e750b5850fcc\") " pod="openshift-ingress-canary/ingress-canary-pqp48" Jul 23 00:08:41.838581 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:41.838471 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx5gg\" (UniqueName: \"kubernetes.io/projected/dd67adef-fce4-429b-b01b-8225f9c33ab4-kube-api-access-gx5gg\") pod \"dns-default-ndkl6\" (UID: \"dd67adef-fce4-429b-b01b-8225f9c33ab4\") " pod="openshift-dns/dns-default-ndkl6" Jul 23 00:08:41.838581 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:41.838506 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq72q\" (UniqueName: \"kubernetes.io/projected/c1bab695-9ea2-4498-bf9e-e750b5850fcc-kube-api-access-bq72q\") pod \"ingress-canary-pqp48\" (UID: \"c1bab695-9ea2-4498-bf9e-e750b5850fcc\") " pod="openshift-ingress-canary/ingress-canary-pqp48" Jul 23 00:08:41.838581 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:41.838540 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 00:08:41.838847 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:41.838614 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd67adef-fce4-429b-b01b-8225f9c33ab4-metrics-tls podName:dd67adef-fce4-429b-b01b-8225f9c33ab4 nodeName:}" failed. No retries permitted until 2026-07-23 00:08:42.338593913 +0000 UTC m=+36.134100755 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dd67adef-fce4-429b-b01b-8225f9c33ab4-metrics-tls") pod "dns-default-ndkl6" (UID: "dd67adef-fce4-429b-b01b-8225f9c33ab4") : secret "dns-default-metrics-tls" not found Jul 23 00:08:41.838847 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:41.838544 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd67adef-fce4-429b-b01b-8225f9c33ab4-tmp-dir\") pod \"dns-default-ndkl6\" (UID: \"dd67adef-fce4-429b-b01b-8225f9c33ab4\") " pod="openshift-dns/dns-default-ndkl6" Jul 23 00:08:41.838974 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:41.838853 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd67adef-fce4-429b-b01b-8225f9c33ab4-tmp-dir\") pod \"dns-default-ndkl6\" (UID: \"dd67adef-fce4-429b-b01b-8225f9c33ab4\") " pod="openshift-dns/dns-default-ndkl6" Jul 23 00:08:41.839028 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:41.838996 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dd67adef-fce4-429b-b01b-8225f9c33ab4-config-volume\") pod \"dns-default-ndkl6\" (UID: \"dd67adef-fce4-429b-b01b-8225f9c33ab4\") " pod="openshift-dns/dns-default-ndkl6" Jul 23 00:08:41.843277 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:41.843245 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c1bab695-9ea2-4498-bf9e-e750b5850fcc-cert\") pod \"ingress-canary-pqp48\" (UID: \"c1bab695-9ea2-4498-bf9e-e750b5850fcc\") " pod="openshift-ingress-canary/ingress-canary-pqp48" Jul 23 00:08:41.850015 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:41.849984 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx5gg\" (UniqueName: \"kubernetes.io/projected/dd67adef-fce4-429b-b01b-8225f9c33ab4-kube-api-access-gx5gg\") pod \"dns-default-ndkl6\" (UID: \"dd67adef-fce4-429b-b01b-8225f9c33ab4\") " pod="openshift-dns/dns-default-ndkl6" Jul 23 00:08:41.850160 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:41.850031 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq72q\" (UniqueName: \"kubernetes.io/projected/c1bab695-9ea2-4498-bf9e-e750b5850fcc-kube-api-access-bq72q\") pod \"ingress-canary-pqp48\" (UID: \"c1bab695-9ea2-4498-bf9e-e750b5850fcc\") " pod="openshift-ingress-canary/ingress-canary-pqp48" Jul 23 00:08:41.942243 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:41.942201 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pqp48" Jul 23 00:08:42.001529 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:42.001494 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfgzz" Jul 23 00:08:42.001722 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:42.001536 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jlsd6" Jul 23 00:08:42.005144 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:42.004946 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 23 00:08:42.005532 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:42.004954 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 23 00:08:42.005532 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:42.004973 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pff25\"" Jul 23 00:08:42.342306 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:42.342262 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dd67adef-fce4-429b-b01b-8225f9c33ab4-metrics-tls\") pod \"dns-default-ndkl6\" (UID: \"dd67adef-fce4-429b-b01b-8225f9c33ab4\") " pod="openshift-dns/dns-default-ndkl6" Jul 23 00:08:42.342488 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:42.342424 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 00:08:42.342536 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:42.342492 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd67adef-fce4-429b-b01b-8225f9c33ab4-metrics-tls podName:dd67adef-fce4-429b-b01b-8225f9c33ab4 nodeName:}" failed. No retries permitted until 2026-07-23 00:08:43.342474888 +0000 UTC m=+37.137981726 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dd67adef-fce4-429b-b01b-8225f9c33ab4-metrics-tls") pod "dns-default-ndkl6" (UID: "dd67adef-fce4-429b-b01b-8225f9c33ab4") : secret "dns-default-metrics-tls" not found Jul 23 00:08:42.580496 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:42.580462 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pqp48"] Jul 23 00:08:42.659363 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:08:42.659270 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1bab695_9ea2_4498_bf9e_e750b5850fcc.slice/crio-81b23bc2b62b00e045dac7ed680fe1028d6f1454609b88c72e244847ec9e1e29 WatchSource:0}: Error finding container 81b23bc2b62b00e045dac7ed680fe1028d6f1454609b88c72e244847ec9e1e29: Status 404 returned error can't find the container with id 81b23bc2b62b00e045dac7ed680fe1028d6f1454609b88c72e244847ec9e1e29 Jul 23 00:08:43.001428 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:43.001323 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cgvgd" Jul 23 00:08:43.005313 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:43.004954 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 23 00:08:43.006081 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:43.006061 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-d8bbv\"" Jul 23 00:08:43.006185 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:43.006118 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 23 00:08:43.179744 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:43.179701 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pqp48" event={"ID":"c1bab695-9ea2-4498-bf9e-e750b5850fcc","Type":"ContainerStarted","Data":"81b23bc2b62b00e045dac7ed680fe1028d6f1454609b88c72e244847ec9e1e29"} Jul 23 00:08:43.183132 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:43.183088 2562 generic.go:356] "Generic (PLEG): container finished" podID="216d84cd-f540-4bc0-9bd1-a8cc9e37367f" containerID="048fdcbfb1f3624f7a24f69892d911d881e042c3b1c0e95c502189915b5f241c" exitCode=0 Jul 23 00:08:43.183319 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:43.183148 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8ghxf" event={"ID":"216d84cd-f540-4bc0-9bd1-a8cc9e37367f","Type":"ContainerDied","Data":"048fdcbfb1f3624f7a24f69892d911d881e042c3b1c0e95c502189915b5f241c"} Jul 23 00:08:43.350644 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:43.350598 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dd67adef-fce4-429b-b01b-8225f9c33ab4-metrics-tls\") pod \"dns-default-ndkl6\" (UID: \"dd67adef-fce4-429b-b01b-8225f9c33ab4\") " pod="openshift-dns/dns-default-ndkl6" Jul 23 00:08:43.350848 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:43.350717 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 00:08:43.350848 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:43.350816 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd67adef-fce4-429b-b01b-8225f9c33ab4-metrics-tls podName:dd67adef-fce4-429b-b01b-8225f9c33ab4 nodeName:}" failed. No retries permitted until 2026-07-23 00:08:45.350792898 +0000 UTC m=+39.146299756 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dd67adef-fce4-429b-b01b-8225f9c33ab4-metrics-tls") pod "dns-default-ndkl6" (UID: "dd67adef-fce4-429b-b01b-8225f9c33ab4") : secret "dns-default-metrics-tls" not found Jul 23 00:08:44.188377 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:44.188173 2562 generic.go:356] "Generic (PLEG): container finished" podID="216d84cd-f540-4bc0-9bd1-a8cc9e37367f" containerID="eb792c228d85aadc704ff188963ef6c81f387266ede4538bd5c4cc78f80148e2" exitCode=0 Jul 23 00:08:44.188865 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:44.188236 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8ghxf" event={"ID":"216d84cd-f540-4bc0-9bd1-a8cc9e37367f","Type":"ContainerDied","Data":"eb792c228d85aadc704ff188963ef6c81f387266ede4538bd5c4cc78f80148e2"} Jul 23 00:08:45.192660 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:45.192567 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pqp48" event={"ID":"c1bab695-9ea2-4498-bf9e-e750b5850fcc","Type":"ContainerStarted","Data":"bd4edea3c8b2065a08319d5178edb99a2c0bae8c4fa3c27ae8a56eebd26d587b"} Jul 23 00:08:45.196063 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:45.196039 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8ghxf" event={"ID":"216d84cd-f540-4bc0-9bd1-a8cc9e37367f","Type":"ContainerStarted","Data":"ebf35cc6623fc3ec6c5e41cbcca91eafce94033706e7ed4f715817276ef25d31"} Jul 23 00:08:45.209099 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:45.209049 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-pqp48" podStartSLOduration=1.950608158 podStartE2EDuration="4.209032811s" totalImagesPullingTime="2.258424653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:08:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:08:45.208223895 +0000 UTC m=+39.003730753" watchObservedRunningTime="2026-07-23 00:08:45.209032811 +0000 UTC m=+39.004539649" Jul 23 00:08:45.232819 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:45.232733 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-8ghxf" podStartSLOduration=9.554965286 podStartE2EDuration="39.232713904s" totalImagesPullingTime="29.677748618s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:08:06 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:08:45.22862028 +0000 UTC m=+39.024127137" watchObservedRunningTime="2026-07-23 00:08:45.232713904 +0000 UTC m=+39.028220757" Jul 23 00:08:45.362675 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:45.362624 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dd67adef-fce4-429b-b01b-8225f9c33ab4-metrics-tls\") pod \"dns-default-ndkl6\" (UID: \"dd67adef-fce4-429b-b01b-8225f9c33ab4\") " pod="openshift-dns/dns-default-ndkl6" Jul 23 00:08:45.362891 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:45.362850 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 00:08:45.362962 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:45.362934 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd67adef-fce4-429b-b01b-8225f9c33ab4-metrics-tls podName:dd67adef-fce4-429b-b01b-8225f9c33ab4 nodeName:}" failed. No retries permitted until 2026-07-23 00:08:49.362912935 +0000 UTC m=+43.158419777 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dd67adef-fce4-429b-b01b-8225f9c33ab4-metrics-tls") pod "dns-default-ndkl6" (UID: "dd67adef-fce4-429b-b01b-8225f9c33ab4") : secret "dns-default-metrics-tls" not found Jul 23 00:08:49.392512 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:49.392461 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dd67adef-fce4-429b-b01b-8225f9c33ab4-metrics-tls\") pod \"dns-default-ndkl6\" (UID: \"dd67adef-fce4-429b-b01b-8225f9c33ab4\") " pod="openshift-dns/dns-default-ndkl6" Jul 23 00:08:49.393081 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:49.392612 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 00:08:49.393081 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:49.392679 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd67adef-fce4-429b-b01b-8225f9c33ab4-metrics-tls podName:dd67adef-fce4-429b-b01b-8225f9c33ab4 nodeName:}" failed. No retries permitted until 2026-07-23 00:08:57.392663704 +0000 UTC m=+51.188170540 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dd67adef-fce4-429b-b01b-8225f9c33ab4-metrics-tls") pod "dns-default-ndkl6" (UID: "dd67adef-fce4-429b-b01b-8225f9c33ab4") : secret "dns-default-metrics-tls" not found Jul 23 00:08:57.450862 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:08:57.450813 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dd67adef-fce4-429b-b01b-8225f9c33ab4-metrics-tls\") pod \"dns-default-ndkl6\" (UID: \"dd67adef-fce4-429b-b01b-8225f9c33ab4\") " pod="openshift-dns/dns-default-ndkl6" Jul 23 00:08:57.451351 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:57.450949 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 00:08:57.451351 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:08:57.451016 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd67adef-fce4-429b-b01b-8225f9c33ab4-metrics-tls podName:dd67adef-fce4-429b-b01b-8225f9c33ab4 nodeName:}" failed. No retries permitted until 2026-07-23 00:09:13.451001956 +0000 UTC m=+67.246508792 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dd67adef-fce4-429b-b01b-8225f9c33ab4-metrics-tls") pod "dns-default-ndkl6" (UID: "dd67adef-fce4-429b-b01b-8225f9c33ab4") : secret "dns-default-metrics-tls" not found Jul 23 00:09:06.168143 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:09:06.168114 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bjc8c" Jul 23 00:09:12.465400 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:09:12.465358 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f272157-a81d-439c-86ed-511e11146698-original-pull-secret\") pod \"global-pull-secret-syncer-jlsd6\" (UID: \"5f272157-a81d-439c-86ed-511e11146698\") " pod="kube-system/global-pull-secret-syncer-jlsd6" Jul 23 00:09:12.465400 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:09:12.465403 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a9bdee1-a493-4223-a7b6-3ae1f5024eb6-metrics-certs\") pod \"network-metrics-daemon-kfgzz\" (UID: \"5a9bdee1-a493-4223-a7b6-3ae1f5024eb6\") " pod="openshift-multus/network-metrics-daemon-kfgzz" Jul 23 00:09:12.468454 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:09:12.468436 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 23 00:09:12.468613 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:09:12.468590 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 23 00:09:12.476689 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:09:12.476662 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 23 00:09:12.476825 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:09:12.476738 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a9bdee1-a493-4223-a7b6-3ae1f5024eb6-metrics-certs podName:5a9bdee1-a493-4223-a7b6-3ae1f5024eb6 nodeName:}" failed. No retries permitted until 2026-07-23 00:10:16.476718514 +0000 UTC m=+130.272225352 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5a9bdee1-a493-4223-a7b6-3ae1f5024eb6-metrics-certs") pod "network-metrics-daemon-kfgzz" (UID: "5a9bdee1-a493-4223-a7b6-3ae1f5024eb6") : secret "metrics-daemon-secret" not found Jul 23 00:09:12.480493 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:09:12.480472 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f272157-a81d-439c-86ed-511e11146698-original-pull-secret\") pod \"global-pull-secret-syncer-jlsd6\" (UID: \"5f272157-a81d-439c-86ed-511e11146698\") " pod="kube-system/global-pull-secret-syncer-jlsd6" Jul 23 00:09:12.566381 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:09:12.566340 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2thdl\" (UniqueName: \"kubernetes.io/projected/72c0b00d-f59f-4186-b84e-470a1adac941-kube-api-access-2thdl\") pod \"network-check-target-cgvgd\" (UID: \"72c0b00d-f59f-4186-b84e-470a1adac941\") " pod="openshift-network-diagnostics/network-check-target-cgvgd" Jul 23 00:09:12.569430 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:09:12.569404 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 23 00:09:12.579080 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:09:12.579048 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 23 00:09:12.589684 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:09:12.589656 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2thdl\" (UniqueName: \"kubernetes.io/projected/72c0b00d-f59f-4186-b84e-470a1adac941-kube-api-access-2thdl\") pod \"network-check-target-cgvgd\" (UID: \"72c0b00d-f59f-4186-b84e-470a1adac941\") " pod="openshift-network-diagnostics/network-check-target-cgvgd" Jul 23 00:09:12.621738 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:09:12.621694 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jlsd6" Jul 23 00:09:12.715722 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:09:12.715643 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-d8bbv\"" Jul 23 00:09:12.723862 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:09:12.723826 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cgvgd" Jul 23 00:09:12.749717 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:09:12.749680 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jlsd6"] Jul 23 00:09:12.753007 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:09:12.752974 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f272157_a81d_439c_86ed_511e11146698.slice/crio-05c9b2331cb3e41b7c679984700bc1665e353ac681d8c568daa4c25b6b61614a WatchSource:0}: Error finding container 05c9b2331cb3e41b7c679984700bc1665e353ac681d8c568daa4c25b6b61614a: Status 404 returned error can't find the container with id 05c9b2331cb3e41b7c679984700bc1665e353ac681d8c568daa4c25b6b61614a Jul 23 00:09:12.852041 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:09:12.852007 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cgvgd"] Jul 23 00:09:12.855256 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:09:12.855215 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72c0b00d_f59f_4186_b84e_470a1adac941.slice/crio-c718ce187df80f1745a35bfa57dd20c050e97ab2e7903f8a5c637e31a5777c44 WatchSource:0}: Error finding container c718ce187df80f1745a35bfa57dd20c050e97ab2e7903f8a5c637e31a5777c44: Status 404 returned error can't find the container with id c718ce187df80f1745a35bfa57dd20c050e97ab2e7903f8a5c637e31a5777c44 Jul 23 00:09:13.260832 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:09:13.260789 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cgvgd" event={"ID":"72c0b00d-f59f-4186-b84e-470a1adac941","Type":"ContainerStarted","Data":"c718ce187df80f1745a35bfa57dd20c050e97ab2e7903f8a5c637e31a5777c44"} Jul 23 00:09:13.262156 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:09:13.262116 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jlsd6" event={"ID":"5f272157-a81d-439c-86ed-511e11146698","Type":"ContainerStarted","Data":"05c9b2331cb3e41b7c679984700bc1665e353ac681d8c568daa4c25b6b61614a"} Jul 23 00:09:13.473295 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:09:13.473081 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dd67adef-fce4-429b-b01b-8225f9c33ab4-metrics-tls\") pod \"dns-default-ndkl6\" (UID: \"dd67adef-fce4-429b-b01b-8225f9c33ab4\") " pod="openshift-dns/dns-default-ndkl6" Jul 23 00:09:13.473295 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:09:13.473237 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 00:09:13.473295 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:09:13.473305 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd67adef-fce4-429b-b01b-8225f9c33ab4-metrics-tls podName:dd67adef-fce4-429b-b01b-8225f9c33ab4 nodeName:}" failed. No retries permitted until 2026-07-23 00:09:45.473285242 +0000 UTC m=+99.268792091 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dd67adef-fce4-429b-b01b-8225f9c33ab4-metrics-tls") pod "dns-default-ndkl6" (UID: "dd67adef-fce4-429b-b01b-8225f9c33ab4") : secret "dns-default-metrics-tls" not found Jul 23 00:09:18.279030 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:09:18.278986 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jlsd6" event={"ID":"5f272157-a81d-439c-86ed-511e11146698","Type":"ContainerStarted","Data":"696dc6c0c13228fb86ffd4894d284066e9e3b93dffbe34b5a98289aba2735e53"} Jul 23 00:09:18.280554 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:09:18.280526 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cgvgd" event={"ID":"72c0b00d-f59f-4186-b84e-470a1adac941","Type":"ContainerStarted","Data":"15204e5270a3c4e03206ce8e40d9f43cbb3ab8443518c79e0632e5142b59f649"} Jul 23 00:09:18.280675 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:09:18.280663 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-cgvgd" Jul 23 00:09:18.294862 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:09:18.294812 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-jlsd6" podStartSLOduration=66.266453979 podStartE2EDuration="1m11.294797314s" totalImagesPullingTime="5.028343335s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:08:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:09:18.29396052 +0000 UTC m=+72.089467376" watchObservedRunningTime="2026-07-23 00:09:18.294797314 +0000 UTC m=+72.090304171" Jul 23 00:09:18.308649 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:09:18.308593 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-cgvgd" podStartSLOduration=66.394305219 podStartE2EDuration="1m11.308575669s" totalImagesPullingTime="4.91427045s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:08:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:09:18.308535328 +0000 UTC m=+72.104042185" watchObservedRunningTime="2026-07-23 00:09:18.308575669 +0000 UTC m=+72.104082586" Jul 23 00:09:45.503431 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:09:45.503378 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dd67adef-fce4-429b-b01b-8225f9c33ab4-metrics-tls\") pod \"dns-default-ndkl6\" (UID: \"dd67adef-fce4-429b-b01b-8225f9c33ab4\") " pod="openshift-dns/dns-default-ndkl6" Jul 23 00:09:45.504028 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:09:45.503538 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 00:09:45.504028 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:09:45.503621 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd67adef-fce4-429b-b01b-8225f9c33ab4-metrics-tls podName:dd67adef-fce4-429b-b01b-8225f9c33ab4 nodeName:}" failed. No retries permitted until 2026-07-23 00:10:49.503599183 +0000 UTC m=+163.299106020 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dd67adef-fce4-429b-b01b-8225f9c33ab4-metrics-tls") pod "dns-default-ndkl6" (UID: "dd67adef-fce4-429b-b01b-8225f9c33ab4") : secret "dns-default-metrics-tls" not found Jul 23 00:09:49.286152 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:09:49.286120 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-cgvgd" Jul 23 00:10:16.520242 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:10:16.520180 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a9bdee1-a493-4223-a7b6-3ae1f5024eb6-metrics-certs\") pod \"network-metrics-daemon-kfgzz\" (UID: \"5a9bdee1-a493-4223-a7b6-3ae1f5024eb6\") " pod="openshift-multus/network-metrics-daemon-kfgzz" Jul 23 00:10:16.520786 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:10:16.520344 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 23 00:10:16.520786 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:10:16.520420 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a9bdee1-a493-4223-a7b6-3ae1f5024eb6-metrics-certs podName:5a9bdee1-a493-4223-a7b6-3ae1f5024eb6 nodeName:}" failed. No retries permitted until 2026-07-23 00:12:18.520400217 +0000 UTC m=+252.315907052 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5a9bdee1-a493-4223-a7b6-3ae1f5024eb6-metrics-certs") pod "network-metrics-daemon-kfgzz" (UID: "5a9bdee1-a493-4223-a7b6-3ae1f5024eb6") : secret "metrics-daemon-secret" not found Jul 23 00:10:44.623507 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:10:44.623458 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-ndkl6" podUID="dd67adef-fce4-429b-b01b-8225f9c33ab4" Jul 23 00:10:45.015071 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:10:45.014976 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-kfgzz" podUID="5a9bdee1-a493-4223-a7b6-3ae1f5024eb6" Jul 23 00:10:45.486433 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:10:45.486401 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ndkl6" Jul 23 00:10:49.556392 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:10:49.556337 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dd67adef-fce4-429b-b01b-8225f9c33ab4-metrics-tls\") pod \"dns-default-ndkl6\" (UID: \"dd67adef-fce4-429b-b01b-8225f9c33ab4\") " pod="openshift-dns/dns-default-ndkl6" Jul 23 00:10:49.556868 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:10:49.556468 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 00:10:49.556868 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:10:49.556532 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd67adef-fce4-429b-b01b-8225f9c33ab4-metrics-tls podName:dd67adef-fce4-429b-b01b-8225f9c33ab4 nodeName:}" failed. No retries permitted until 2026-07-23 00:12:51.556518018 +0000 UTC m=+285.352024853 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dd67adef-fce4-429b-b01b-8225f9c33ab4-metrics-tls") pod "dns-default-ndkl6" (UID: "dd67adef-fce4-429b-b01b-8225f9c33ab4") : secret "dns-default-metrics-tls" not found Jul 23 00:10:57.002136 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:10:57.002094 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfgzz" Jul 23 00:12:01.440477 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.440442 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-2dnw6"] Jul 23 00:12:01.442106 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.442086 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-555d9d7885-w24mg"] Jul 23 00:12:01.442227 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.442208 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-2dnw6" Jul 23 00:12:01.443673 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.443655 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-w24mg" Jul 23 00:12:01.445773 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.445741 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-jtlkn\"" Jul 23 00:12:01.446344 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.446317 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 23 00:12:01.447076 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.447057 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 23 00:12:01.447280 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.447256 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-qb87x\"" Jul 23 00:12:01.447421 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.447300 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 23 00:12:01.447421 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.447311 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 23 00:12:01.449479 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.449454 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-lzd8m"] Jul 23 00:12:01.451257 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.451230 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-lzd8m" Jul 23 00:12:01.452843 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.452821 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 23 00:12:01.453980 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.453963 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 23 00:12:01.454036 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.454005 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 23 00:12:01.454808 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.454794 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-5xfp8\"" Jul 23 00:12:01.454903 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.454884 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 23 00:12:01.462648 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.462620 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-2dnw6"] Jul 23 00:12:01.463726 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.463701 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-w24mg"] Jul 23 00:12:01.469191 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.469164 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-lzd8m"] Jul 23 00:12:01.536315 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.536278 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-nqcdd"] Jul 23 00:12:01.538336 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.538316 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-nqcdd" Jul 23 00:12:01.540899 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.540865 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 23 00:12:01.541041 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.540943 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-q6w22\"" Jul 23 00:12:01.541041 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.540975 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 23 00:12:01.544989 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.544879 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-pkvw6"] Jul 23 00:12:01.547088 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.547068 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-76974999cb-mggnm"] Jul 23 00:12:01.547199 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.547182 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-pkvw6" Jul 23 00:12:01.549168 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.549148 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-76974999cb-mggnm" Jul 23 00:12:01.549856 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.549827 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 23 00:12:01.549963 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.549894 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-cww5l\"" Jul 23 00:12:01.550145 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.550124 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-nqcdd"] Jul 23 00:12:01.550265 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.550250 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 23 00:12:01.550505 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.550487 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 23 00:12:01.550623 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.550509 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 23 00:12:01.551668 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.551647 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 23 00:12:01.552223 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.552201 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 23 00:12:01.552491 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.552470 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 23 00:12:01.552575 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.552523 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-qt5sz\"" Jul 23 00:12:01.552575 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.552535 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ec2a61a7-0a82-4f9a-aac7-001dec5f5a6f-snapshots\") pod \"insights-operator-555d9d7885-w24mg\" (UID: \"ec2a61a7-0a82-4f9a-aac7-001dec5f5a6f\") " pod="openshift-insights/insights-operator-555d9d7885-w24mg" Jul 23 00:12:01.552575 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.552566 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec2a61a7-0a82-4f9a-aac7-001dec5f5a6f-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-w24mg\" (UID: \"ec2a61a7-0a82-4f9a-aac7-001dec5f5a6f\") " pod="openshift-insights/insights-operator-555d9d7885-w24mg" Jul 23 00:12:01.552715 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.552473 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 23 00:12:01.552715 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.552606 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grthl\" (UniqueName: \"kubernetes.io/projected/ec2a61a7-0a82-4f9a-aac7-001dec5f5a6f-kube-api-access-grthl\") pod \"insights-operator-555d9d7885-w24mg\" (UID: \"ec2a61a7-0a82-4f9a-aac7-001dec5f5a6f\") " pod="openshift-insights/insights-operator-555d9d7885-w24mg" Jul 23 00:12:01.552715 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.552642 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcnmt\" (UniqueName: \"kubernetes.io/projected/5bbb2725-2f3b-4b41-a586-4d36c72f29f6-kube-api-access-qcnmt\") pod \"network-check-source-7bc5b6bf85-2dnw6\" (UID: \"5bbb2725-2f3b-4b41-a586-4d36c72f29f6\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-2dnw6" Jul 23 00:12:01.552715 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.552677 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ec2a61a7-0a82-4f9a-aac7-001dec5f5a6f-serving-cert\") pod \"insights-operator-555d9d7885-w24mg\" (UID: \"ec2a61a7-0a82-4f9a-aac7-001dec5f5a6f\") " pod="openshift-insights/insights-operator-555d9d7885-w24mg" Jul 23 00:12:01.552937 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.552780 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec2a61a7-0a82-4f9a-aac7-001dec5f5a6f-service-ca-bundle\") pod \"insights-operator-555d9d7885-w24mg\" (UID: \"ec2a61a7-0a82-4f9a-aac7-001dec5f5a6f\") " pod="openshift-insights/insights-operator-555d9d7885-w24mg" Jul 23 00:12:01.552937 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.552827 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh824\" (UniqueName: \"kubernetes.io/projected/1ad6b737-50ee-45f5-9bf7-d95f619e8e82-kube-api-access-nh824\") pod \"cluster-samples-operator-79b7c55f87-lzd8m\" (UID: \"1ad6b737-50ee-45f5-9bf7-d95f619e8e82\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-lzd8m" Jul 23 00:12:01.552937 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.552884 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec2a61a7-0a82-4f9a-aac7-001dec5f5a6f-tmp\") pod \"insights-operator-555d9d7885-w24mg\" (UID: \"ec2a61a7-0a82-4f9a-aac7-001dec5f5a6f\") " pod="openshift-insights/insights-operator-555d9d7885-w24mg" Jul 23 00:12:01.553090 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.552949 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 23 00:12:01.553090 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.552968 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ad6b737-50ee-45f5-9bf7-d95f619e8e82-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-lzd8m\" (UID: \"1ad6b737-50ee-45f5-9bf7-d95f619e8e82\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-lzd8m" Jul 23 00:12:01.553172 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.553106 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 23 00:12:01.559181 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.559154 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-pkvw6"] Jul 23 00:12:01.562559 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.562533 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-76974999cb-mggnm"] Jul 23 00:12:01.653654 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.653586 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh824\" (UniqueName: \"kubernetes.io/projected/1ad6b737-50ee-45f5-9bf7-d95f619e8e82-kube-api-access-nh824\") pod \"cluster-samples-operator-79b7c55f87-lzd8m\" (UID: \"1ad6b737-50ee-45f5-9bf7-d95f619e8e82\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-lzd8m" Jul 23 00:12:01.653654 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.653659 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec2a61a7-0a82-4f9a-aac7-001dec5f5a6f-tmp\") pod \"insights-operator-555d9d7885-w24mg\" (UID: \"ec2a61a7-0a82-4f9a-aac7-001dec5f5a6f\") " pod="openshift-insights/insights-operator-555d9d7885-w24mg" Jul 23 00:12:01.653953 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.653687 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ad6b737-50ee-45f5-9bf7-d95f619e8e82-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-lzd8m\" (UID: \"1ad6b737-50ee-45f5-9bf7-d95f619e8e82\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-lzd8m" Jul 23 00:12:01.653953 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.653716 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5hkr\" (UniqueName: \"kubernetes.io/projected/107a082e-c6f4-4d8d-a55a-763f1bedc830-kube-api-access-l5hkr\") pod \"volume-data-source-validator-78cf4c6f6-nqcdd\" (UID: \"107a082e-c6f4-4d8d-a55a-763f1bedc830\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-nqcdd" Jul 23 00:12:01.653953 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.653740 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/190fa13e-db15-4ace-91ed-7d1fd190fe6c-config\") pod \"service-ca-operator-5bcbc94c78-pkvw6\" (UID: \"190fa13e-db15-4ace-91ed-7d1fd190fe6c\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-pkvw6" Jul 23 00:12:01.653953 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.653826 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ec2a61a7-0a82-4f9a-aac7-001dec5f5a6f-snapshots\") pod \"insights-operator-555d9d7885-w24mg\" (UID: \"ec2a61a7-0a82-4f9a-aac7-001dec5f5a6f\") " pod="openshift-insights/insights-operator-555d9d7885-w24mg" Jul 23 00:12:01.653953 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.653861 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec2a61a7-0a82-4f9a-aac7-001dec5f5a6f-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-w24mg\" (UID: \"ec2a61a7-0a82-4f9a-aac7-001dec5f5a6f\") " pod="openshift-insights/insights-operator-555d9d7885-w24mg" Jul 23 00:12:01.653953 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:12:01.653869 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 23 00:12:01.653953 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.653906 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grthl\" (UniqueName: \"kubernetes.io/projected/ec2a61a7-0a82-4f9a-aac7-001dec5f5a6f-kube-api-access-grthl\") pod \"insights-operator-555d9d7885-w24mg\" (UID: \"ec2a61a7-0a82-4f9a-aac7-001dec5f5a6f\") " pod="openshift-insights/insights-operator-555d9d7885-w24mg" Jul 23 00:12:01.653953 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:12:01.653929 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ad6b737-50ee-45f5-9bf7-d95f619e8e82-samples-operator-tls podName:1ad6b737-50ee-45f5-9bf7-d95f619e8e82 nodeName:}" failed. No retries permitted until 2026-07-23 00:12:02.153910531 +0000 UTC m=+235.949417386 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1ad6b737-50ee-45f5-9bf7-d95f619e8e82-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-lzd8m" (UID: "1ad6b737-50ee-45f5-9bf7-d95f619e8e82") : secret "samples-operator-tls" not found Jul 23 00:12:01.654305 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.653958 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bkrp\" (UniqueName: \"kubernetes.io/projected/c0ba4e0d-2b3e-4a20-8f3e-1296830be034-kube-api-access-2bkrp\") pod \"router-default-76974999cb-mggnm\" (UID: \"c0ba4e0d-2b3e-4a20-8f3e-1296830be034\") " pod="openshift-ingress/router-default-76974999cb-mggnm" Jul 23 00:12:01.654305 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.653990 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0ba4e0d-2b3e-4a20-8f3e-1296830be034-service-ca-bundle\") pod \"router-default-76974999cb-mggnm\" (UID: \"c0ba4e0d-2b3e-4a20-8f3e-1296830be034\") " pod="openshift-ingress/router-default-76974999cb-mggnm" Jul 23 00:12:01.654305 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.654042 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcnmt\" (UniqueName: \"kubernetes.io/projected/5bbb2725-2f3b-4b41-a586-4d36c72f29f6-kube-api-access-qcnmt\") pod \"network-check-source-7bc5b6bf85-2dnw6\" (UID: \"5bbb2725-2f3b-4b41-a586-4d36c72f29f6\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-2dnw6" Jul 23 00:12:01.654305 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.654093 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ec2a61a7-0a82-4f9a-aac7-001dec5f5a6f-serving-cert\") pod \"insights-operator-555d9d7885-w24mg\" (UID: \"ec2a61a7-0a82-4f9a-aac7-001dec5f5a6f\") " pod="openshift-insights/insights-operator-555d9d7885-w24mg" Jul 23 00:12:01.654305 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.654127 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/190fa13e-db15-4ace-91ed-7d1fd190fe6c-serving-cert\") pod \"service-ca-operator-5bcbc94c78-pkvw6\" (UID: \"190fa13e-db15-4ace-91ed-7d1fd190fe6c\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-pkvw6" Jul 23 00:12:01.654305 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.654153 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c0ba4e0d-2b3e-4a20-8f3e-1296830be034-stats-auth\") pod \"router-default-76974999cb-mggnm\" (UID: \"c0ba4e0d-2b3e-4a20-8f3e-1296830be034\") " pod="openshift-ingress/router-default-76974999cb-mggnm" Jul 23 00:12:01.654305 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.654182 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0ba4e0d-2b3e-4a20-8f3e-1296830be034-metrics-certs\") pod \"router-default-76974999cb-mggnm\" (UID: \"c0ba4e0d-2b3e-4a20-8f3e-1296830be034\") " pod="openshift-ingress/router-default-76974999cb-mggnm" Jul 23 00:12:01.654305 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.654209 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec2a61a7-0a82-4f9a-aac7-001dec5f5a6f-service-ca-bundle\") pod \"insights-operator-555d9d7885-w24mg\" (UID: \"ec2a61a7-0a82-4f9a-aac7-001dec5f5a6f\") " pod="openshift-insights/insights-operator-555d9d7885-w24mg" Jul 23 00:12:01.654305 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.654234 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw8zd\" (UniqueName: \"kubernetes.io/projected/190fa13e-db15-4ace-91ed-7d1fd190fe6c-kube-api-access-lw8zd\") pod \"service-ca-operator-5bcbc94c78-pkvw6\" (UID: \"190fa13e-db15-4ace-91ed-7d1fd190fe6c\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-pkvw6" Jul 23 00:12:01.654305 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.654245 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec2a61a7-0a82-4f9a-aac7-001dec5f5a6f-tmp\") pod \"insights-operator-555d9d7885-w24mg\" (UID: \"ec2a61a7-0a82-4f9a-aac7-001dec5f5a6f\") " pod="openshift-insights/insights-operator-555d9d7885-w24mg" Jul 23 00:12:01.654305 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.654259 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c0ba4e0d-2b3e-4a20-8f3e-1296830be034-default-certificate\") pod \"router-default-76974999cb-mggnm\" (UID: \"c0ba4e0d-2b3e-4a20-8f3e-1296830be034\") " pod="openshift-ingress/router-default-76974999cb-mggnm" Jul 23 00:12:01.654844 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.654493 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ec2a61a7-0a82-4f9a-aac7-001dec5f5a6f-snapshots\") pod \"insights-operator-555d9d7885-w24mg\" (UID: \"ec2a61a7-0a82-4f9a-aac7-001dec5f5a6f\") " pod="openshift-insights/insights-operator-555d9d7885-w24mg" Jul 23 00:12:01.654844 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.654839 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec2a61a7-0a82-4f9a-aac7-001dec5f5a6f-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-w24mg\" (UID: \"ec2a61a7-0a82-4f9a-aac7-001dec5f5a6f\") " pod="openshift-insights/insights-operator-555d9d7885-w24mg" Jul 23 00:12:01.655060 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.655039 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec2a61a7-0a82-4f9a-aac7-001dec5f5a6f-service-ca-bundle\") pod \"insights-operator-555d9d7885-w24mg\" (UID: \"ec2a61a7-0a82-4f9a-aac7-001dec5f5a6f\") " pod="openshift-insights/insights-operator-555d9d7885-w24mg" Jul 23 00:12:01.656535 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.656518 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ec2a61a7-0a82-4f9a-aac7-001dec5f5a6f-serving-cert\") pod \"insights-operator-555d9d7885-w24mg\" (UID: \"ec2a61a7-0a82-4f9a-aac7-001dec5f5a6f\") " pod="openshift-insights/insights-operator-555d9d7885-w24mg" Jul 23 00:12:01.662451 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.662419 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh824\" (UniqueName: \"kubernetes.io/projected/1ad6b737-50ee-45f5-9bf7-d95f619e8e82-kube-api-access-nh824\") pod \"cluster-samples-operator-79b7c55f87-lzd8m\" (UID: \"1ad6b737-50ee-45f5-9bf7-d95f619e8e82\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-lzd8m" Jul 23 00:12:01.662597 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.662535 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grthl\" (UniqueName: \"kubernetes.io/projected/ec2a61a7-0a82-4f9a-aac7-001dec5f5a6f-kube-api-access-grthl\") pod \"insights-operator-555d9d7885-w24mg\" (UID: \"ec2a61a7-0a82-4f9a-aac7-001dec5f5a6f\") " pod="openshift-insights/insights-operator-555d9d7885-w24mg" Jul 23 00:12:01.662597 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.662568 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcnmt\" (UniqueName: \"kubernetes.io/projected/5bbb2725-2f3b-4b41-a586-4d36c72f29f6-kube-api-access-qcnmt\") pod \"network-check-source-7bc5b6bf85-2dnw6\" (UID: \"5bbb2725-2f3b-4b41-a586-4d36c72f29f6\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-2dnw6" Jul 23 00:12:01.753338 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.753222 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-2dnw6" Jul 23 00:12:01.754727 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.754697 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0ba4e0d-2b3e-4a20-8f3e-1296830be034-metrics-certs\") pod \"router-default-76974999cb-mggnm\" (UID: \"c0ba4e0d-2b3e-4a20-8f3e-1296830be034\") " pod="openshift-ingress/router-default-76974999cb-mggnm" Jul 23 00:12:01.754848 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.754744 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw8zd\" (UniqueName: \"kubernetes.io/projected/190fa13e-db15-4ace-91ed-7d1fd190fe6c-kube-api-access-lw8zd\") pod \"service-ca-operator-5bcbc94c78-pkvw6\" (UID: \"190fa13e-db15-4ace-91ed-7d1fd190fe6c\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-pkvw6" Jul 23 00:12:01.754848 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.754795 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c0ba4e0d-2b3e-4a20-8f3e-1296830be034-default-certificate\") pod \"router-default-76974999cb-mggnm\" (UID: \"c0ba4e0d-2b3e-4a20-8f3e-1296830be034\") " pod="openshift-ingress/router-default-76974999cb-mggnm" Jul 23 00:12:01.754964 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:12:01.754855 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 23 00:12:01.754964 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.754859 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5hkr\" (UniqueName: \"kubernetes.io/projected/107a082e-c6f4-4d8d-a55a-763f1bedc830-kube-api-access-l5hkr\") pod \"volume-data-source-validator-78cf4c6f6-nqcdd\" (UID: \"107a082e-c6f4-4d8d-a55a-763f1bedc830\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-nqcdd" Jul 23 00:12:01.754964 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:12:01.754917 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0ba4e0d-2b3e-4a20-8f3e-1296830be034-metrics-certs podName:c0ba4e0d-2b3e-4a20-8f3e-1296830be034 nodeName:}" failed. No retries permitted until 2026-07-23 00:12:02.254896354 +0000 UTC m=+236.050403202 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0ba4e0d-2b3e-4a20-8f3e-1296830be034-metrics-certs") pod "router-default-76974999cb-mggnm" (UID: "c0ba4e0d-2b3e-4a20-8f3e-1296830be034") : secret "router-metrics-certs-default" not found Jul 23 00:12:01.754964 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.754943 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/190fa13e-db15-4ace-91ed-7d1fd190fe6c-config\") pod \"service-ca-operator-5bcbc94c78-pkvw6\" (UID: \"190fa13e-db15-4ace-91ed-7d1fd190fe6c\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-pkvw6" Jul 23 00:12:01.755358 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.755000 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bkrp\" (UniqueName: \"kubernetes.io/projected/c0ba4e0d-2b3e-4a20-8f3e-1296830be034-kube-api-access-2bkrp\") pod \"router-default-76974999cb-mggnm\" (UID: \"c0ba4e0d-2b3e-4a20-8f3e-1296830be034\") " pod="openshift-ingress/router-default-76974999cb-mggnm" Jul 23 00:12:01.755358 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.755022 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0ba4e0d-2b3e-4a20-8f3e-1296830be034-service-ca-bundle\") pod \"router-default-76974999cb-mggnm\" (UID: \"c0ba4e0d-2b3e-4a20-8f3e-1296830be034\") " pod="openshift-ingress/router-default-76974999cb-mggnm" Jul 23 00:12:01.755358 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.755067 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/190fa13e-db15-4ace-91ed-7d1fd190fe6c-serving-cert\") pod \"service-ca-operator-5bcbc94c78-pkvw6\" (UID: \"190fa13e-db15-4ace-91ed-7d1fd190fe6c\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-pkvw6" Jul 23 00:12:01.755358 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.755090 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c0ba4e0d-2b3e-4a20-8f3e-1296830be034-stats-auth\") pod \"router-default-76974999cb-mggnm\" (UID: \"c0ba4e0d-2b3e-4a20-8f3e-1296830be034\") " pod="openshift-ingress/router-default-76974999cb-mggnm" Jul 23 00:12:01.755358 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:12:01.755173 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c0ba4e0d-2b3e-4a20-8f3e-1296830be034-service-ca-bundle podName:c0ba4e0d-2b3e-4a20-8f3e-1296830be034 nodeName:}" failed. No retries permitted until 2026-07-23 00:12:02.25515396 +0000 UTC m=+236.050660812 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c0ba4e0d-2b3e-4a20-8f3e-1296830be034-service-ca-bundle") pod "router-default-76974999cb-mggnm" (UID: "c0ba4e0d-2b3e-4a20-8f3e-1296830be034") : configmap references non-existent config key: service-ca.crt Jul 23 00:12:01.755616 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.755560 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/190fa13e-db15-4ace-91ed-7d1fd190fe6c-config\") pod \"service-ca-operator-5bcbc94c78-pkvw6\" (UID: \"190fa13e-db15-4ace-91ed-7d1fd190fe6c\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-pkvw6" Jul 23 00:12:01.757349 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.757317 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/190fa13e-db15-4ace-91ed-7d1fd190fe6c-serving-cert\") pod \"service-ca-operator-5bcbc94c78-pkvw6\" (UID: \"190fa13e-db15-4ace-91ed-7d1fd190fe6c\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-pkvw6" Jul 23 00:12:01.757440 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.757407 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c0ba4e0d-2b3e-4a20-8f3e-1296830be034-stats-auth\") pod \"router-default-76974999cb-mggnm\" (UID: \"c0ba4e0d-2b3e-4a20-8f3e-1296830be034\") " pod="openshift-ingress/router-default-76974999cb-mggnm" Jul 23 00:12:01.757485 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.757465 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c0ba4e0d-2b3e-4a20-8f3e-1296830be034-default-certificate\") pod \"router-default-76974999cb-mggnm\" (UID: \"c0ba4e0d-2b3e-4a20-8f3e-1296830be034\") " pod="openshift-ingress/router-default-76974999cb-mggnm" Jul 23 00:12:01.759933 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.759911 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-w24mg" Jul 23 00:12:01.764249 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.764223 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw8zd\" (UniqueName: \"kubernetes.io/projected/190fa13e-db15-4ace-91ed-7d1fd190fe6c-kube-api-access-lw8zd\") pod \"service-ca-operator-5bcbc94c78-pkvw6\" (UID: \"190fa13e-db15-4ace-91ed-7d1fd190fe6c\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-pkvw6" Jul 23 00:12:01.764454 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.764432 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bkrp\" (UniqueName: \"kubernetes.io/projected/c0ba4e0d-2b3e-4a20-8f3e-1296830be034-kube-api-access-2bkrp\") pod \"router-default-76974999cb-mggnm\" (UID: \"c0ba4e0d-2b3e-4a20-8f3e-1296830be034\") " pod="openshift-ingress/router-default-76974999cb-mggnm" Jul 23 00:12:01.764501 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.764427 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5hkr\" (UniqueName: \"kubernetes.io/projected/107a082e-c6f4-4d8d-a55a-763f1bedc830-kube-api-access-l5hkr\") pod \"volume-data-source-validator-78cf4c6f6-nqcdd\" (UID: \"107a082e-c6f4-4d8d-a55a-763f1bedc830\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-nqcdd" Jul 23 00:12:01.848265 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.848225 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-nqcdd" Jul 23 00:12:01.857914 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.857841 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-pkvw6" Jul 23 00:12:01.895092 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.895034 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-2dnw6"] Jul 23 00:12:01.898236 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:12:01.898171 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bbb2725_2f3b_4b41_a586_4d36c72f29f6.slice/crio-7fb804d87e645e2e8daf7948c7348e2079ad7fbdb4d3d3e51bc721b996a00e24 WatchSource:0}: Error finding container 7fb804d87e645e2e8daf7948c7348e2079ad7fbdb4d3d3e51bc721b996a00e24: Status 404 returned error can't find the container with id 7fb804d87e645e2e8daf7948c7348e2079ad7fbdb4d3d3e51bc721b996a00e24 Jul 23 00:12:01.914026 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:01.913948 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-w24mg"] Jul 23 00:12:02.001775 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:02.001737 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-nqcdd"] Jul 23 00:12:02.005672 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:12:02.005637 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod107a082e_c6f4_4d8d_a55a_763f1bedc830.slice/crio-87a84a5dce2a8d946779b6a77f9599ff40994f19f3696d4fc7d6961d8020db50 WatchSource:0}: Error finding container 87a84a5dce2a8d946779b6a77f9599ff40994f19f3696d4fc7d6961d8020db50: Status 404 returned error can't find the container with id 87a84a5dce2a8d946779b6a77f9599ff40994f19f3696d4fc7d6961d8020db50 Jul 23 00:12:02.017093 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:02.017067 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-pkvw6"] Jul 23 00:12:02.020946 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:12:02.020914 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod190fa13e_db15_4ace_91ed_7d1fd190fe6c.slice/crio-f8421f925e50d67314d382030afe3562ef788b889a0b100f4a277ec4453f8daf WatchSource:0}: Error finding container f8421f925e50d67314d382030afe3562ef788b889a0b100f4a277ec4453f8daf: Status 404 returned error can't find the container with id f8421f925e50d67314d382030afe3562ef788b889a0b100f4a277ec4453f8daf Jul 23 00:12:02.159077 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:02.159041 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ad6b737-50ee-45f5-9bf7-d95f619e8e82-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-lzd8m\" (UID: \"1ad6b737-50ee-45f5-9bf7-d95f619e8e82\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-lzd8m" Jul 23 00:12:02.159302 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:12:02.159209 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 23 00:12:02.159302 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:12:02.159281 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ad6b737-50ee-45f5-9bf7-d95f619e8e82-samples-operator-tls podName:1ad6b737-50ee-45f5-9bf7-d95f619e8e82 nodeName:}" failed. No retries permitted until 2026-07-23 00:12:03.159265691 +0000 UTC m=+236.954772529 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1ad6b737-50ee-45f5-9bf7-d95f619e8e82-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-lzd8m" (UID: "1ad6b737-50ee-45f5-9bf7-d95f619e8e82") : secret "samples-operator-tls" not found Jul 23 00:12:02.260004 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:02.259908 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0ba4e0d-2b3e-4a20-8f3e-1296830be034-service-ca-bundle\") pod \"router-default-76974999cb-mggnm\" (UID: \"c0ba4e0d-2b3e-4a20-8f3e-1296830be034\") " pod="openshift-ingress/router-default-76974999cb-mggnm" Jul 23 00:12:02.260004 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:02.259957 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0ba4e0d-2b3e-4a20-8f3e-1296830be034-metrics-certs\") pod \"router-default-76974999cb-mggnm\" (UID: \"c0ba4e0d-2b3e-4a20-8f3e-1296830be034\") " pod="openshift-ingress/router-default-76974999cb-mggnm" Jul 23 00:12:02.260226 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:12:02.260050 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 23 00:12:02.260226 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:12:02.260074 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c0ba4e0d-2b3e-4a20-8f3e-1296830be034-service-ca-bundle podName:c0ba4e0d-2b3e-4a20-8f3e-1296830be034 nodeName:}" failed. No retries permitted until 2026-07-23 00:12:03.260057204 +0000 UTC m=+237.055564039 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c0ba4e0d-2b3e-4a20-8f3e-1296830be034-service-ca-bundle") pod "router-default-76974999cb-mggnm" (UID: "c0ba4e0d-2b3e-4a20-8f3e-1296830be034") : configmap references non-existent config key: service-ca.crt Jul 23 00:12:02.260226 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:12:02.260100 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0ba4e0d-2b3e-4a20-8f3e-1296830be034-metrics-certs podName:c0ba4e0d-2b3e-4a20-8f3e-1296830be034 nodeName:}" failed. No retries permitted until 2026-07-23 00:12:03.260093919 +0000 UTC m=+237.055600754 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0ba4e0d-2b3e-4a20-8f3e-1296830be034-metrics-certs") pod "router-default-76974999cb-mggnm" (UID: "c0ba4e0d-2b3e-4a20-8f3e-1296830be034") : secret "router-metrics-certs-default" not found Jul 23 00:12:02.670185 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:02.670112 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-pkvw6" event={"ID":"190fa13e-db15-4ace-91ed-7d1fd190fe6c","Type":"ContainerStarted","Data":"f8421f925e50d67314d382030afe3562ef788b889a0b100f4a277ec4453f8daf"} Jul 23 00:12:02.672015 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:02.671979 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-w24mg" event={"ID":"ec2a61a7-0a82-4f9a-aac7-001dec5f5a6f","Type":"ContainerStarted","Data":"f0bffc5429f3e0c1a782e690ddf261e9ae093c9ffa026991582fbe18a09c081e"} Jul 23 00:12:02.677219 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:02.673884 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-2dnw6" event={"ID":"5bbb2725-2f3b-4b41-a586-4d36c72f29f6","Type":"ContainerStarted","Data":"a3b98293dea25c8475289aade1796d3e18f3792a9ec27d85855a94d5ae7878d3"} Jul 23 00:12:02.677219 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:02.673938 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-2dnw6" event={"ID":"5bbb2725-2f3b-4b41-a586-4d36c72f29f6","Type":"ContainerStarted","Data":"7fb804d87e645e2e8daf7948c7348e2079ad7fbdb4d3d3e51bc721b996a00e24"} Jul 23 00:12:02.677449 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:02.677370 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-nqcdd" event={"ID":"107a082e-c6f4-4d8d-a55a-763f1bedc830","Type":"ContainerStarted","Data":"87a84a5dce2a8d946779b6a77f9599ff40994f19f3696d4fc7d6961d8020db50"} Jul 23 00:12:02.690369 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:02.690277 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-2dnw6" podStartSLOduration=1.69025692 podStartE2EDuration="1.69025692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:12:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:12:02.69009749 +0000 UTC m=+236.485604351" watchObservedRunningTime="2026-07-23 00:12:02.69025692 +0000 UTC m=+236.485763777" Jul 23 00:12:03.169866 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:03.169824 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ad6b737-50ee-45f5-9bf7-d95f619e8e82-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-lzd8m\" (UID: \"1ad6b737-50ee-45f5-9bf7-d95f619e8e82\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-lzd8m" Jul 23 00:12:03.170045 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:12:03.169975 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 23 00:12:03.170084 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:12:03.170046 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ad6b737-50ee-45f5-9bf7-d95f619e8e82-samples-operator-tls podName:1ad6b737-50ee-45f5-9bf7-d95f619e8e82 nodeName:}" failed. No retries permitted until 2026-07-23 00:12:05.170030667 +0000 UTC m=+238.965537502 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1ad6b737-50ee-45f5-9bf7-d95f619e8e82-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-lzd8m" (UID: "1ad6b737-50ee-45f5-9bf7-d95f619e8e82") : secret "samples-operator-tls" not found Jul 23 00:12:03.271001 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:03.270963 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0ba4e0d-2b3e-4a20-8f3e-1296830be034-metrics-certs\") pod \"router-default-76974999cb-mggnm\" (UID: \"c0ba4e0d-2b3e-4a20-8f3e-1296830be034\") " pod="openshift-ingress/router-default-76974999cb-mggnm" Jul 23 00:12:03.271207 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:03.271098 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0ba4e0d-2b3e-4a20-8f3e-1296830be034-service-ca-bundle\") pod \"router-default-76974999cb-mggnm\" (UID: \"c0ba4e0d-2b3e-4a20-8f3e-1296830be034\") " pod="openshift-ingress/router-default-76974999cb-mggnm" Jul 23 00:12:03.271207 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:12:03.271147 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 23 00:12:03.271338 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:12:03.271229 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0ba4e0d-2b3e-4a20-8f3e-1296830be034-metrics-certs podName:c0ba4e0d-2b3e-4a20-8f3e-1296830be034 nodeName:}" failed. No retries permitted until 2026-07-23 00:12:05.271206787 +0000 UTC m=+239.066713628 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0ba4e0d-2b3e-4a20-8f3e-1296830be034-metrics-certs") pod "router-default-76974999cb-mggnm" (UID: "c0ba4e0d-2b3e-4a20-8f3e-1296830be034") : secret "router-metrics-certs-default" not found Jul 23 00:12:03.271338 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:12:03.271269 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c0ba4e0d-2b3e-4a20-8f3e-1296830be034-service-ca-bundle podName:c0ba4e0d-2b3e-4a20-8f3e-1296830be034 nodeName:}" failed. No retries permitted until 2026-07-23 00:12:05.271258395 +0000 UTC m=+239.066765245 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c0ba4e0d-2b3e-4a20-8f3e-1296830be034-service-ca-bundle") pod "router-default-76974999cb-mggnm" (UID: "c0ba4e0d-2b3e-4a20-8f3e-1296830be034") : configmap references non-existent config key: service-ca.crt Jul 23 00:12:03.682474 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:03.682428 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-nqcdd" event={"ID":"107a082e-c6f4-4d8d-a55a-763f1bedc830","Type":"ContainerStarted","Data":"29842e7dc820820917caecde6bf5d20d5184c047f61861a874c3382765fec4a9"} Jul 23 00:12:03.697776 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:03.697676 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-nqcdd" podStartSLOduration=1.5231702980000001 podStartE2EDuration="2.697653458s" totalImagesPullingTime="1.17448316s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:12:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:12:03.696476154 +0000 UTC m=+237.491983012" watchObservedRunningTime="2026-07-23 00:12:03.697653458 +0000 UTC m=+237.493160317" Jul 23 00:12:04.687730 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:04.687689 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-w24mg" event={"ID":"ec2a61a7-0a82-4f9a-aac7-001dec5f5a6f","Type":"ContainerStarted","Data":"593b715fa2847d8812e2fc16736414eb6eff8f974fbf416ab66e50c53678ad8d"} Jul 23 00:12:04.704153 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:04.704088 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-555d9d7885-w24mg" podStartSLOduration=1.3711624439999999 podStartE2EDuration="3.704065175s" totalImagesPullingTime="2.332902731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:12:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:12:04.703415876 +0000 UTC m=+238.498922734" watchObservedRunningTime="2026-07-23 00:12:04.704065175 +0000 UTC m=+238.499572036" Jul 23 00:12:04.904350 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:04.904310 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-nz9r9"] Jul 23 00:12:04.909051 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:04.909025 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nz9r9" Jul 23 00:12:04.912908 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:04.912220 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5btmf\"" Jul 23 00:12:04.912908 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:04.912480 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 23 00:12:04.912908 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:04.912722 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 23 00:12:04.924722 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:04.924688 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nz9r9"] Jul 23 00:12:04.988086 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:04.987993 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/462cefbf-6678-4aa7-96ba-66a0630f1a63-crio-socket\") pod \"insights-runtime-extractor-nz9r9\" (UID: \"462cefbf-6678-4aa7-96ba-66a0630f1a63\") " pod="openshift-insights/insights-runtime-extractor-nz9r9" Jul 23 00:12:04.988247 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:04.988123 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5fsx\" (UniqueName: \"kubernetes.io/projected/462cefbf-6678-4aa7-96ba-66a0630f1a63-kube-api-access-h5fsx\") pod \"insights-runtime-extractor-nz9r9\" (UID: \"462cefbf-6678-4aa7-96ba-66a0630f1a63\") " pod="openshift-insights/insights-runtime-extractor-nz9r9" Jul 23 00:12:04.988247 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:04.988172 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/462cefbf-6678-4aa7-96ba-66a0630f1a63-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nz9r9\" (UID: \"462cefbf-6678-4aa7-96ba-66a0630f1a63\") " pod="openshift-insights/insights-runtime-extractor-nz9r9" Jul 23 00:12:04.988247 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:04.988211 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/462cefbf-6678-4aa7-96ba-66a0630f1a63-data-volume\") pod \"insights-runtime-extractor-nz9r9\" (UID: \"462cefbf-6678-4aa7-96ba-66a0630f1a63\") " pod="openshift-insights/insights-runtime-extractor-nz9r9" Jul 23 00:12:04.988396 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:04.988302 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/462cefbf-6678-4aa7-96ba-66a0630f1a63-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nz9r9\" (UID: \"462cefbf-6678-4aa7-96ba-66a0630f1a63\") " pod="openshift-insights/insights-runtime-extractor-nz9r9" Jul 23 00:12:05.089400 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:05.089372 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5fsx\" (UniqueName: \"kubernetes.io/projected/462cefbf-6678-4aa7-96ba-66a0630f1a63-kube-api-access-h5fsx\") pod \"insights-runtime-extractor-nz9r9\" (UID: \"462cefbf-6678-4aa7-96ba-66a0630f1a63\") " pod="openshift-insights/insights-runtime-extractor-nz9r9" Jul 23 00:12:05.089503 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:05.089418 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/462cefbf-6678-4aa7-96ba-66a0630f1a63-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nz9r9\" (UID: \"462cefbf-6678-4aa7-96ba-66a0630f1a63\") " pod="openshift-insights/insights-runtime-extractor-nz9r9" Jul 23 00:12:05.089562 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:12:05.089528 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 00:12:05.089616 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:12:05.089596 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/462cefbf-6678-4aa7-96ba-66a0630f1a63-insights-runtime-extractor-tls podName:462cefbf-6678-4aa7-96ba-66a0630f1a63 nodeName:}" failed. No retries permitted until 2026-07-23 00:12:05.589576814 +0000 UTC m=+239.385083663 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/462cefbf-6678-4aa7-96ba-66a0630f1a63-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nz9r9" (UID: "462cefbf-6678-4aa7-96ba-66a0630f1a63") : secret "insights-runtime-extractor-tls" not found Jul 23 00:12:05.089679 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:05.089616 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/462cefbf-6678-4aa7-96ba-66a0630f1a63-data-volume\") pod \"insights-runtime-extractor-nz9r9\" (UID: \"462cefbf-6678-4aa7-96ba-66a0630f1a63\") " pod="openshift-insights/insights-runtime-extractor-nz9r9" Jul 23 00:12:05.089734 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:05.089682 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/462cefbf-6678-4aa7-96ba-66a0630f1a63-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nz9r9\" (UID: \"462cefbf-6678-4aa7-96ba-66a0630f1a63\") " pod="openshift-insights/insights-runtime-extractor-nz9r9" Jul 23 00:12:05.089817 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:05.089747 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/462cefbf-6678-4aa7-96ba-66a0630f1a63-crio-socket\") pod \"insights-runtime-extractor-nz9r9\" (UID: \"462cefbf-6678-4aa7-96ba-66a0630f1a63\") " pod="openshift-insights/insights-runtime-extractor-nz9r9" Jul 23 00:12:05.089859 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:12:05.089830 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 00:12:05.089921 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:12:05.089907 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/462cefbf-6678-4aa7-96ba-66a0630f1a63-kube-rbac-proxy-cm podName:462cefbf-6678-4aa7-96ba-66a0630f1a63 nodeName:}" failed. No retries permitted until 2026-07-23 00:12:05.589890531 +0000 UTC m=+239.385397380 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/462cefbf-6678-4aa7-96ba-66a0630f1a63-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nz9r9" (UID: "462cefbf-6678-4aa7-96ba-66a0630f1a63") : configmap "kube-rbac-proxy" not found Jul 23 00:12:05.089967 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:05.089834 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/462cefbf-6678-4aa7-96ba-66a0630f1a63-crio-socket\") pod \"insights-runtime-extractor-nz9r9\" (UID: \"462cefbf-6678-4aa7-96ba-66a0630f1a63\") " pod="openshift-insights/insights-runtime-extractor-nz9r9" Jul 23 00:12:05.090011 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:05.089994 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/462cefbf-6678-4aa7-96ba-66a0630f1a63-data-volume\") pod \"insights-runtime-extractor-nz9r9\" (UID: \"462cefbf-6678-4aa7-96ba-66a0630f1a63\") " pod="openshift-insights/insights-runtime-extractor-nz9r9" Jul 23 00:12:05.099173 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:05.099145 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5fsx\" (UniqueName: \"kubernetes.io/projected/462cefbf-6678-4aa7-96ba-66a0630f1a63-kube-api-access-h5fsx\") pod \"insights-runtime-extractor-nz9r9\" (UID: \"462cefbf-6678-4aa7-96ba-66a0630f1a63\") " pod="openshift-insights/insights-runtime-extractor-nz9r9" Jul 23 00:12:05.191083 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:05.191036 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ad6b737-50ee-45f5-9bf7-d95f619e8e82-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-lzd8m\" (UID: \"1ad6b737-50ee-45f5-9bf7-d95f619e8e82\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-lzd8m" Jul 23 00:12:05.191269 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:12:05.191191 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 23 00:12:05.191269 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:12:05.191267 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ad6b737-50ee-45f5-9bf7-d95f619e8e82-samples-operator-tls podName:1ad6b737-50ee-45f5-9bf7-d95f619e8e82 nodeName:}" failed. No retries permitted until 2026-07-23 00:12:09.191241628 +0000 UTC m=+242.986748476 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1ad6b737-50ee-45f5-9bf7-d95f619e8e82-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-lzd8m" (UID: "1ad6b737-50ee-45f5-9bf7-d95f619e8e82") : secret "samples-operator-tls" not found Jul 23 00:12:05.292358 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:05.292314 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0ba4e0d-2b3e-4a20-8f3e-1296830be034-metrics-certs\") pod \"router-default-76974999cb-mggnm\" (UID: \"c0ba4e0d-2b3e-4a20-8f3e-1296830be034\") " pod="openshift-ingress/router-default-76974999cb-mggnm" Jul 23 00:12:05.292538 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:05.292414 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0ba4e0d-2b3e-4a20-8f3e-1296830be034-service-ca-bundle\") pod \"router-default-76974999cb-mggnm\" (UID: \"c0ba4e0d-2b3e-4a20-8f3e-1296830be034\") " pod="openshift-ingress/router-default-76974999cb-mggnm" Jul 23 00:12:05.292538 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:12:05.292473 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 23 00:12:05.292538 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:12:05.292537 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0ba4e0d-2b3e-4a20-8f3e-1296830be034-metrics-certs podName:c0ba4e0d-2b3e-4a20-8f3e-1296830be034 nodeName:}" failed. No retries permitted until 2026-07-23 00:12:09.292521576 +0000 UTC m=+243.088028424 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0ba4e0d-2b3e-4a20-8f3e-1296830be034-metrics-certs") pod "router-default-76974999cb-mggnm" (UID: "c0ba4e0d-2b3e-4a20-8f3e-1296830be034") : secret "router-metrics-certs-default" not found Jul 23 00:12:05.292651 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:12:05.292596 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c0ba4e0d-2b3e-4a20-8f3e-1296830be034-service-ca-bundle podName:c0ba4e0d-2b3e-4a20-8f3e-1296830be034 nodeName:}" failed. No retries permitted until 2026-07-23 00:12:09.292575335 +0000 UTC m=+243.088082177 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c0ba4e0d-2b3e-4a20-8f3e-1296830be034-service-ca-bundle") pod "router-default-76974999cb-mggnm" (UID: "c0ba4e0d-2b3e-4a20-8f3e-1296830be034") : configmap references non-existent config key: service-ca.crt Jul 23 00:12:05.595303 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:05.595200 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/462cefbf-6678-4aa7-96ba-66a0630f1a63-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nz9r9\" (UID: \"462cefbf-6678-4aa7-96ba-66a0630f1a63\") " pod="openshift-insights/insights-runtime-extractor-nz9r9" Jul 23 00:12:05.595303 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:05.595289 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/462cefbf-6678-4aa7-96ba-66a0630f1a63-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nz9r9\" (UID: \"462cefbf-6678-4aa7-96ba-66a0630f1a63\") " pod="openshift-insights/insights-runtime-extractor-nz9r9" Jul 23 00:12:05.595493 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:12:05.595368 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 00:12:05.595493 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:12:05.595381 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 00:12:05.595493 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:12:05.595434 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/462cefbf-6678-4aa7-96ba-66a0630f1a63-kube-rbac-proxy-cm podName:462cefbf-6678-4aa7-96ba-66a0630f1a63 nodeName:}" failed. No retries permitted until 2026-07-23 00:12:06.595419417 +0000 UTC m=+240.390926252 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/462cefbf-6678-4aa7-96ba-66a0630f1a63-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nz9r9" (UID: "462cefbf-6678-4aa7-96ba-66a0630f1a63") : configmap "kube-rbac-proxy" not found Jul 23 00:12:05.595493 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:12:05.595448 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/462cefbf-6678-4aa7-96ba-66a0630f1a63-insights-runtime-extractor-tls podName:462cefbf-6678-4aa7-96ba-66a0630f1a63 nodeName:}" failed. No retries permitted until 2026-07-23 00:12:06.595443406 +0000 UTC m=+240.390950242 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/462cefbf-6678-4aa7-96ba-66a0630f1a63-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nz9r9" (UID: "462cefbf-6678-4aa7-96ba-66a0630f1a63") : secret "insights-runtime-extractor-tls" not found Jul 23 00:12:05.692777 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:05.692709 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-pkvw6" event={"ID":"190fa13e-db15-4ace-91ed-7d1fd190fe6c","Type":"ContainerStarted","Data":"87440c9e4fff04803e43bde32227f394d9bbe62c8028d1266909c9fdb83fc975"} Jul 23 00:12:05.709049 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:05.708990 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-pkvw6" podStartSLOduration=1.6443584740000001 podStartE2EDuration="4.708974229s" totalImagesPullingTime="3.064615755s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:12:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:12:05.708054111 +0000 UTC m=+239.503560967" watchObservedRunningTime="2026-07-23 00:12:05.708974229 +0000 UTC m=+239.504481086" Jul 23 00:12:06.603951 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:06.603913 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/462cefbf-6678-4aa7-96ba-66a0630f1a63-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nz9r9\" (UID: \"462cefbf-6678-4aa7-96ba-66a0630f1a63\") " pod="openshift-insights/insights-runtime-extractor-nz9r9" Jul 23 00:12:06.604173 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:06.603975 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/462cefbf-6678-4aa7-96ba-66a0630f1a63-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nz9r9\" (UID: \"462cefbf-6678-4aa7-96ba-66a0630f1a63\") " pod="openshift-insights/insights-runtime-extractor-nz9r9" Jul 23 00:12:06.604173 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:12:06.604046 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 00:12:06.604173 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:12:06.604082 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 00:12:06.604173 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:12:06.604096 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/462cefbf-6678-4aa7-96ba-66a0630f1a63-kube-rbac-proxy-cm podName:462cefbf-6678-4aa7-96ba-66a0630f1a63 nodeName:}" failed. No retries permitted until 2026-07-23 00:12:08.604083178 +0000 UTC m=+242.399590012 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/462cefbf-6678-4aa7-96ba-66a0630f1a63-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nz9r9" (UID: "462cefbf-6678-4aa7-96ba-66a0630f1a63") : configmap "kube-rbac-proxy" not found Jul 23 00:12:06.604173 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:12:06.604173 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/462cefbf-6678-4aa7-96ba-66a0630f1a63-insights-runtime-extractor-tls podName:462cefbf-6678-4aa7-96ba-66a0630f1a63 nodeName:}" failed. No retries permitted until 2026-07-23 00:12:08.604155497 +0000 UTC m=+242.399662350 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/462cefbf-6678-4aa7-96ba-66a0630f1a63-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nz9r9" (UID: "462cefbf-6678-4aa7-96ba-66a0630f1a63") : secret "insights-runtime-extractor-tls" not found Jul 23 00:12:06.650857 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:06.650826 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9stwt_7d457c50-6bca-4609-8897-cef340b087e8/dns-node-resolver/0.log" Jul 23 00:12:06.875735 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:06.875655 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-bs2ws"] Jul 23 00:12:06.878731 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:06.878713 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-bs2ws" Jul 23 00:12:06.881701 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:06.881672 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-j649x\"" Jul 23 00:12:06.881701 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:06.881672 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 23 00:12:06.881897 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:06.881708 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 23 00:12:06.888521 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:06.888502 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-bs2ws"] Jul 23 00:12:07.007077 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:07.007052 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqmfx\" (UniqueName: \"kubernetes.io/projected/1b62b19c-4feb-48ef-807e-37741fa4db0b-kube-api-access-sqmfx\") pod \"migrator-7cc9c7ffd7-bs2ws\" (UID: \"1b62b19c-4feb-48ef-807e-37741fa4db0b\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-bs2ws" Jul 23 00:12:07.107434 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:07.107398 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqmfx\" (UniqueName: \"kubernetes.io/projected/1b62b19c-4feb-48ef-807e-37741fa4db0b-kube-api-access-sqmfx\") pod \"migrator-7cc9c7ffd7-bs2ws\" (UID: \"1b62b19c-4feb-48ef-807e-37741fa4db0b\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-bs2ws" Jul 23 00:12:07.117999 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:07.117972 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqmfx\" (UniqueName: \"kubernetes.io/projected/1b62b19c-4feb-48ef-807e-37741fa4db0b-kube-api-access-sqmfx\") pod \"migrator-7cc9c7ffd7-bs2ws\" (UID: \"1b62b19c-4feb-48ef-807e-37741fa4db0b\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-bs2ws" Jul 23 00:12:07.189033 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:07.188940 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-bs2ws" Jul 23 00:12:07.322665 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:07.322630 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-bs2ws"] Jul 23 00:12:07.325550 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:12:07.325511 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b62b19c_4feb_48ef_807e_37741fa4db0b.slice/crio-0f73be57bf4123767e5876d978954345634c8809fe94682dc9fbf74cac198f2c WatchSource:0}: Error finding container 0f73be57bf4123767e5876d978954345634c8809fe94682dc9fbf74cac198f2c: Status 404 returned error can't find the container with id 0f73be57bf4123767e5876d978954345634c8809fe94682dc9fbf74cac198f2c Jul 23 00:12:07.699422 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:07.699386 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-bs2ws" event={"ID":"1b62b19c-4feb-48ef-807e-37741fa4db0b","Type":"ContainerStarted","Data":"0f73be57bf4123767e5876d978954345634c8809fe94682dc9fbf74cac198f2c"} Jul 23 00:12:07.851052 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:07.851017 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jctrc_aeaeb6df-4c50-464f-b5fa-a6e599d2fdf9/node-ca/0.log" Jul 23 00:12:08.618656 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:08.618613 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/462cefbf-6678-4aa7-96ba-66a0630f1a63-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nz9r9\" (UID: \"462cefbf-6678-4aa7-96ba-66a0630f1a63\") " pod="openshift-insights/insights-runtime-extractor-nz9r9" Jul 23 00:12:08.619058 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:08.618680 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/462cefbf-6678-4aa7-96ba-66a0630f1a63-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nz9r9\" (UID: \"462cefbf-6678-4aa7-96ba-66a0630f1a63\") " pod="openshift-insights/insights-runtime-extractor-nz9r9" Jul 23 00:12:08.619058 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:12:08.618779 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 00:12:08.619058 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:12:08.618790 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 00:12:08.619058 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:12:08.618842 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/462cefbf-6678-4aa7-96ba-66a0630f1a63-kube-rbac-proxy-cm podName:462cefbf-6678-4aa7-96ba-66a0630f1a63 nodeName:}" failed. No retries permitted until 2026-07-23 00:12:12.618827198 +0000 UTC m=+246.414334036 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/462cefbf-6678-4aa7-96ba-66a0630f1a63-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nz9r9" (UID: "462cefbf-6678-4aa7-96ba-66a0630f1a63") : configmap "kube-rbac-proxy" not found Jul 23 00:12:08.619058 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:12:08.618856 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/462cefbf-6678-4aa7-96ba-66a0630f1a63-insights-runtime-extractor-tls podName:462cefbf-6678-4aa7-96ba-66a0630f1a63 nodeName:}" failed. No retries permitted until 2026-07-23 00:12:12.618849627 +0000 UTC m=+246.414356462 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/462cefbf-6678-4aa7-96ba-66a0630f1a63-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nz9r9" (UID: "462cefbf-6678-4aa7-96ba-66a0630f1a63") : secret "insights-runtime-extractor-tls" not found Jul 23 00:12:08.650383 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:08.650352 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pqp48_c1bab695-9ea2-4498-bf9e-e750b5850fcc/serve-healthcheck-canary/0.log" Jul 23 00:12:08.704723 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:08.704639 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-bs2ws" event={"ID":"1b62b19c-4feb-48ef-807e-37741fa4db0b","Type":"ContainerStarted","Data":"a923d18e7564a1afea917d1435fd58c04602c6fdf792b14fe4871766ec3172b4"} Jul 23 00:12:08.704723 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:08.704691 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-bs2ws" event={"ID":"1b62b19c-4feb-48ef-807e-37741fa4db0b","Type":"ContainerStarted","Data":"b177957b9befae23c23939c8388ed85d78058d7548c1d664db8a6995e884c9ba"} Jul 23 00:12:08.722881 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:08.722826 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-bs2ws" podStartSLOduration=1.631538374 podStartE2EDuration="2.722807873s" totalImagesPullingTime="1.091269499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:12:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:12:08.721738905 +0000 UTC m=+242.517245761" watchObservedRunningTime="2026-07-23 00:12:08.722807873 +0000 UTC m=+242.518314730" Jul 23 00:12:09.224750 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:09.224707 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ad6b737-50ee-45f5-9bf7-d95f619e8e82-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-lzd8m\" (UID: \"1ad6b737-50ee-45f5-9bf7-d95f619e8e82\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-lzd8m" Jul 23 00:12:09.224938 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:12:09.224861 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 23 00:12:09.224938 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:12:09.224928 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ad6b737-50ee-45f5-9bf7-d95f619e8e82-samples-operator-tls podName:1ad6b737-50ee-45f5-9bf7-d95f619e8e82 nodeName:}" failed. No retries permitted until 2026-07-23 00:12:17.224912071 +0000 UTC m=+251.020418906 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1ad6b737-50ee-45f5-9bf7-d95f619e8e82-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-lzd8m" (UID: "1ad6b737-50ee-45f5-9bf7-d95f619e8e82") : secret "samples-operator-tls" not found Jul 23 00:12:09.325567 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:09.325518 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0ba4e0d-2b3e-4a20-8f3e-1296830be034-metrics-certs\") pod \"router-default-76974999cb-mggnm\" (UID: \"c0ba4e0d-2b3e-4a20-8f3e-1296830be034\") " pod="openshift-ingress/router-default-76974999cb-mggnm" Jul 23 00:12:09.325810 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:09.325612 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0ba4e0d-2b3e-4a20-8f3e-1296830be034-service-ca-bundle\") pod \"router-default-76974999cb-mggnm\" (UID: \"c0ba4e0d-2b3e-4a20-8f3e-1296830be034\") " pod="openshift-ingress/router-default-76974999cb-mggnm" Jul 23 00:12:09.325810 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:12:09.325665 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 23 00:12:09.325810 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:12:09.325734 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0ba4e0d-2b3e-4a20-8f3e-1296830be034-metrics-certs podName:c0ba4e0d-2b3e-4a20-8f3e-1296830be034 nodeName:}" failed. No retries permitted until 2026-07-23 00:12:17.325717927 +0000 UTC m=+251.121224766 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0ba4e0d-2b3e-4a20-8f3e-1296830be034-metrics-certs") pod "router-default-76974999cb-mggnm" (UID: "c0ba4e0d-2b3e-4a20-8f3e-1296830be034") : secret "router-metrics-certs-default" not found Jul 23 00:12:09.325810 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:12:09.325749 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c0ba4e0d-2b3e-4a20-8f3e-1296830be034-service-ca-bundle podName:c0ba4e0d-2b3e-4a20-8f3e-1296830be034 nodeName:}" failed. No retries permitted until 2026-07-23 00:12:17.325742407 +0000 UTC m=+251.121249245 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c0ba4e0d-2b3e-4a20-8f3e-1296830be034-service-ca-bundle") pod "router-default-76974999cb-mggnm" (UID: "c0ba4e0d-2b3e-4a20-8f3e-1296830be034") : configmap references non-existent config key: service-ca.crt Jul 23 00:12:12.652121 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:12.652081 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/462cefbf-6678-4aa7-96ba-66a0630f1a63-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nz9r9\" (UID: \"462cefbf-6678-4aa7-96ba-66a0630f1a63\") " pod="openshift-insights/insights-runtime-extractor-nz9r9" Jul 23 00:12:12.652570 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:12.652174 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/462cefbf-6678-4aa7-96ba-66a0630f1a63-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nz9r9\" (UID: \"462cefbf-6678-4aa7-96ba-66a0630f1a63\") " pod="openshift-insights/insights-runtime-extractor-nz9r9" Jul 23 00:12:12.652570 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:12:12.652221 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 00:12:12.652570 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:12:12.652281 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 00:12:12.652570 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:12:12.652311 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/462cefbf-6678-4aa7-96ba-66a0630f1a63-kube-rbac-proxy-cm podName:462cefbf-6678-4aa7-96ba-66a0630f1a63 nodeName:}" failed. No retries permitted until 2026-07-23 00:12:20.652288867 +0000 UTC m=+254.447795706 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/462cefbf-6678-4aa7-96ba-66a0630f1a63-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nz9r9" (UID: "462cefbf-6678-4aa7-96ba-66a0630f1a63") : configmap "kube-rbac-proxy" not found Jul 23 00:12:12.652570 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:12:12.652332 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/462cefbf-6678-4aa7-96ba-66a0630f1a63-insights-runtime-extractor-tls podName:462cefbf-6678-4aa7-96ba-66a0630f1a63 nodeName:}" failed. No retries permitted until 2026-07-23 00:12:20.652322361 +0000 UTC m=+254.447829198 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/462cefbf-6678-4aa7-96ba-66a0630f1a63-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nz9r9" (UID: "462cefbf-6678-4aa7-96ba-66a0630f1a63") : secret "insights-runtime-extractor-tls" not found Jul 23 00:12:17.294016 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:17.293969 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ad6b737-50ee-45f5-9bf7-d95f619e8e82-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-lzd8m\" (UID: \"1ad6b737-50ee-45f5-9bf7-d95f619e8e82\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-lzd8m" Jul 23 00:12:17.296569 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:17.296536 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ad6b737-50ee-45f5-9bf7-d95f619e8e82-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-lzd8m\" (UID: \"1ad6b737-50ee-45f5-9bf7-d95f619e8e82\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-lzd8m" Jul 23 00:12:17.369402 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:17.369375 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-5xfp8\"" Jul 23 00:12:17.376487 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:17.376458 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-lzd8m" Jul 23 00:12:17.395128 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:17.395097 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0ba4e0d-2b3e-4a20-8f3e-1296830be034-metrics-certs\") pod \"router-default-76974999cb-mggnm\" (UID: \"c0ba4e0d-2b3e-4a20-8f3e-1296830be034\") " pod="openshift-ingress/router-default-76974999cb-mggnm" Jul 23 00:12:17.395325 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:17.395160 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0ba4e0d-2b3e-4a20-8f3e-1296830be034-service-ca-bundle\") pod \"router-default-76974999cb-mggnm\" (UID: \"c0ba4e0d-2b3e-4a20-8f3e-1296830be034\") " pod="openshift-ingress/router-default-76974999cb-mggnm" Jul 23 00:12:17.395325 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:12:17.395273 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c0ba4e0d-2b3e-4a20-8f3e-1296830be034-service-ca-bundle podName:c0ba4e0d-2b3e-4a20-8f3e-1296830be034 nodeName:}" failed. No retries permitted until 2026-07-23 00:12:33.395258053 +0000 UTC m=+267.190764888 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c0ba4e0d-2b3e-4a20-8f3e-1296830be034-service-ca-bundle") pod "router-default-76974999cb-mggnm" (UID: "c0ba4e0d-2b3e-4a20-8f3e-1296830be034") : configmap references non-existent config key: service-ca.crt Jul 23 00:12:17.397814 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:17.397785 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0ba4e0d-2b3e-4a20-8f3e-1296830be034-metrics-certs\") pod \"router-default-76974999cb-mggnm\" (UID: \"c0ba4e0d-2b3e-4a20-8f3e-1296830be034\") " pod="openshift-ingress/router-default-76974999cb-mggnm" Jul 23 00:12:17.509108 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:17.509074 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-lzd8m"] Jul 23 00:12:17.734031 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:17.733996 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-lzd8m" event={"ID":"1ad6b737-50ee-45f5-9bf7-d95f619e8e82","Type":"ContainerStarted","Data":"2651b5449d753a261898604fa3ef3ca555ebc45bfdf0e3bf13f2acb34c1364a4"} Jul 23 00:12:18.604544 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:18.604508 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a9bdee1-a493-4223-a7b6-3ae1f5024eb6-metrics-certs\") pod \"network-metrics-daemon-kfgzz\" (UID: \"5a9bdee1-a493-4223-a7b6-3ae1f5024eb6\") " pod="openshift-multus/network-metrics-daemon-kfgzz" Jul 23 00:12:18.605032 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:12:18.604678 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 23 00:12:18.605032 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:12:18.604781 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a9bdee1-a493-4223-a7b6-3ae1f5024eb6-metrics-certs podName:5a9bdee1-a493-4223-a7b6-3ae1f5024eb6 nodeName:}" failed. No retries permitted until 2026-07-23 00:14:20.604741779 +0000 UTC m=+374.400248628 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5a9bdee1-a493-4223-a7b6-3ae1f5024eb6-metrics-certs") pod "network-metrics-daemon-kfgzz" (UID: "5a9bdee1-a493-4223-a7b6-3ae1f5024eb6") : secret "metrics-daemon-secret" not found Jul 23 00:12:19.742285 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:19.742231 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-lzd8m" event={"ID":"1ad6b737-50ee-45f5-9bf7-d95f619e8e82","Type":"ContainerStarted","Data":"f6fbae41a2d9448635444e27d87e750ec487ea9fc812e2a6c6801ddb1e250757"} Jul 23 00:12:19.742709 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:19.742333 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-lzd8m" event={"ID":"1ad6b737-50ee-45f5-9bf7-d95f619e8e82","Type":"ContainerStarted","Data":"02af5e7550525e0a5b23870774376576f7f3ca7d3b08a6441a9700e21f918354"} Jul 23 00:12:19.759125 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:19.759073 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-lzd8m" podStartSLOduration=16.783227709 podStartE2EDuration="18.759056227s" totalImagesPullingTime="1.975828518s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:12:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:12:19.758350919 +0000 UTC m=+253.553857776" watchObservedRunningTime="2026-07-23 00:12:19.759056227 +0000 UTC m=+253.554563083" Jul 23 00:12:20.722842 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:20.722800 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/462cefbf-6678-4aa7-96ba-66a0630f1a63-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nz9r9\" (UID: \"462cefbf-6678-4aa7-96ba-66a0630f1a63\") " pod="openshift-insights/insights-runtime-extractor-nz9r9" Jul 23 00:12:20.723022 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:20.722863 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/462cefbf-6678-4aa7-96ba-66a0630f1a63-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nz9r9\" (UID: \"462cefbf-6678-4aa7-96ba-66a0630f1a63\") " pod="openshift-insights/insights-runtime-extractor-nz9r9" Jul 23 00:12:20.723022 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:12:20.722948 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 00:12:20.723022 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:12:20.723018 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/462cefbf-6678-4aa7-96ba-66a0630f1a63-insights-runtime-extractor-tls podName:462cefbf-6678-4aa7-96ba-66a0630f1a63 nodeName:}" failed. No retries permitted until 2026-07-23 00:12:36.723003298 +0000 UTC m=+270.518510133 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/462cefbf-6678-4aa7-96ba-66a0630f1a63-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nz9r9" (UID: "462cefbf-6678-4aa7-96ba-66a0630f1a63") : secret "insights-runtime-extractor-tls" not found Jul 23 00:12:20.723357 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:20.723341 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/462cefbf-6678-4aa7-96ba-66a0630f1a63-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nz9r9\" (UID: \"462cefbf-6678-4aa7-96ba-66a0630f1a63\") " pod="openshift-insights/insights-runtime-extractor-nz9r9" Jul 23 00:12:29.709649 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:29.709570 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5cc57669fb-7w5wx"] Jul 23 00:12:29.712551 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:29.712530 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cc57669fb-7w5wx" Jul 23 00:12:29.715276 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:29.715241 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 23 00:12:29.715528 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:29.715512 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-t8qkr\"" Jul 23 00:12:29.715576 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:29.715532 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 23 00:12:29.715623 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:29.715591 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 23 00:12:29.722019 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:29.721994 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 23 00:12:29.725601 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:29.725576 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5cc57669fb-7w5wx"] Jul 23 00:12:29.795038 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:29.795000 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1243e526-631f-487f-bcd1-a3a594979839-bound-sa-token\") pod \"image-registry-5cc57669fb-7w5wx\" (UID: \"1243e526-631f-487f-bcd1-a3a594979839\") " pod="openshift-image-registry/image-registry-5cc57669fb-7w5wx" Jul 23 00:12:29.795215 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:29.795040 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1243e526-631f-487f-bcd1-a3a594979839-registry-certificates\") pod \"image-registry-5cc57669fb-7w5wx\" (UID: \"1243e526-631f-487f-bcd1-a3a594979839\") " pod="openshift-image-registry/image-registry-5cc57669fb-7w5wx" Jul 23 00:12:29.795215 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:29.795073 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1243e526-631f-487f-bcd1-a3a594979839-installation-pull-secrets\") pod \"image-registry-5cc57669fb-7w5wx\" (UID: \"1243e526-631f-487f-bcd1-a3a594979839\") " pod="openshift-image-registry/image-registry-5cc57669fb-7w5wx" Jul 23 00:12:29.795215 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:29.795152 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1243e526-631f-487f-bcd1-a3a594979839-ca-trust-extracted\") pod \"image-registry-5cc57669fb-7w5wx\" (UID: \"1243e526-631f-487f-bcd1-a3a594979839\") " pod="openshift-image-registry/image-registry-5cc57669fb-7w5wx" Jul 23 00:12:29.795215 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:29.795183 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1243e526-631f-487f-bcd1-a3a594979839-registry-tls\") pod \"image-registry-5cc57669fb-7w5wx\" (UID: \"1243e526-631f-487f-bcd1-a3a594979839\") " pod="openshift-image-registry/image-registry-5cc57669fb-7w5wx" Jul 23 00:12:29.795215 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:29.795204 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1243e526-631f-487f-bcd1-a3a594979839-image-registry-private-configuration\") pod \"image-registry-5cc57669fb-7w5wx\" (UID: \"1243e526-631f-487f-bcd1-a3a594979839\") " pod="openshift-image-registry/image-registry-5cc57669fb-7w5wx" Jul 23 00:12:29.795403 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:29.795227 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5ssj\" (UniqueName: \"kubernetes.io/projected/1243e526-631f-487f-bcd1-a3a594979839-kube-api-access-v5ssj\") pod \"image-registry-5cc57669fb-7w5wx\" (UID: \"1243e526-631f-487f-bcd1-a3a594979839\") " pod="openshift-image-registry/image-registry-5cc57669fb-7w5wx" Jul 23 00:12:29.795403 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:29.795246 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1243e526-631f-487f-bcd1-a3a594979839-trusted-ca\") pod \"image-registry-5cc57669fb-7w5wx\" (UID: \"1243e526-631f-487f-bcd1-a3a594979839\") " pod="openshift-image-registry/image-registry-5cc57669fb-7w5wx" Jul 23 00:12:29.896035 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:29.895992 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1243e526-631f-487f-bcd1-a3a594979839-ca-trust-extracted\") pod \"image-registry-5cc57669fb-7w5wx\" (UID: \"1243e526-631f-487f-bcd1-a3a594979839\") " pod="openshift-image-registry/image-registry-5cc57669fb-7w5wx" Jul 23 00:12:29.896244 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:29.896037 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1243e526-631f-487f-bcd1-a3a594979839-registry-tls\") pod \"image-registry-5cc57669fb-7w5wx\" (UID: \"1243e526-631f-487f-bcd1-a3a594979839\") " pod="openshift-image-registry/image-registry-5cc57669fb-7w5wx" Jul 23 00:12:29.896244 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:29.896069 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1243e526-631f-487f-bcd1-a3a594979839-image-registry-private-configuration\") pod \"image-registry-5cc57669fb-7w5wx\" (UID: \"1243e526-631f-487f-bcd1-a3a594979839\") " pod="openshift-image-registry/image-registry-5cc57669fb-7w5wx" Jul 23 00:12:29.896244 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:29.896187 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5ssj\" (UniqueName: \"kubernetes.io/projected/1243e526-631f-487f-bcd1-a3a594979839-kube-api-access-v5ssj\") pod \"image-registry-5cc57669fb-7w5wx\" (UID: \"1243e526-631f-487f-bcd1-a3a594979839\") " pod="openshift-image-registry/image-registry-5cc57669fb-7w5wx" Jul 23 00:12:29.896244 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:29.896226 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1243e526-631f-487f-bcd1-a3a594979839-trusted-ca\") pod \"image-registry-5cc57669fb-7w5wx\" (UID: \"1243e526-631f-487f-bcd1-a3a594979839\") " pod="openshift-image-registry/image-registry-5cc57669fb-7w5wx" Jul 23 00:12:29.896459 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:29.896281 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1243e526-631f-487f-bcd1-a3a594979839-bound-sa-token\") pod \"image-registry-5cc57669fb-7w5wx\" (UID: \"1243e526-631f-487f-bcd1-a3a594979839\") " pod="openshift-image-registry/image-registry-5cc57669fb-7w5wx" Jul 23 00:12:29.896459 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:29.896315 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1243e526-631f-487f-bcd1-a3a594979839-registry-certificates\") pod \"image-registry-5cc57669fb-7w5wx\" (UID: \"1243e526-631f-487f-bcd1-a3a594979839\") " pod="openshift-image-registry/image-registry-5cc57669fb-7w5wx" Jul 23 00:12:29.896459 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:29.896347 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1243e526-631f-487f-bcd1-a3a594979839-installation-pull-secrets\") pod \"image-registry-5cc57669fb-7w5wx\" (UID: \"1243e526-631f-487f-bcd1-a3a594979839\") " pod="openshift-image-registry/image-registry-5cc57669fb-7w5wx" Jul 23 00:12:29.896612 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:29.896503 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1243e526-631f-487f-bcd1-a3a594979839-ca-trust-extracted\") pod \"image-registry-5cc57669fb-7w5wx\" (UID: \"1243e526-631f-487f-bcd1-a3a594979839\") " pod="openshift-image-registry/image-registry-5cc57669fb-7w5wx" Jul 23 00:12:29.897128 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:29.897105 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1243e526-631f-487f-bcd1-a3a594979839-registry-certificates\") pod \"image-registry-5cc57669fb-7w5wx\" (UID: \"1243e526-631f-487f-bcd1-a3a594979839\") " pod="openshift-image-registry/image-registry-5cc57669fb-7w5wx" Jul 23 00:12:29.897230 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:29.897164 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1243e526-631f-487f-bcd1-a3a594979839-trusted-ca\") pod \"image-registry-5cc57669fb-7w5wx\" (UID: \"1243e526-631f-487f-bcd1-a3a594979839\") " pod="openshift-image-registry/image-registry-5cc57669fb-7w5wx" Jul 23 00:12:29.898605 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:29.898575 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1243e526-631f-487f-bcd1-a3a594979839-registry-tls\") pod \"image-registry-5cc57669fb-7w5wx\" (UID: \"1243e526-631f-487f-bcd1-a3a594979839\") " pod="openshift-image-registry/image-registry-5cc57669fb-7w5wx" Jul 23 00:12:29.898907 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:29.898890 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1243e526-631f-487f-bcd1-a3a594979839-installation-pull-secrets\") pod \"image-registry-5cc57669fb-7w5wx\" (UID: \"1243e526-631f-487f-bcd1-a3a594979839\") " pod="openshift-image-registry/image-registry-5cc57669fb-7w5wx" Jul 23 00:12:29.899090 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:29.899067 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1243e526-631f-487f-bcd1-a3a594979839-image-registry-private-configuration\") pod \"image-registry-5cc57669fb-7w5wx\" (UID: \"1243e526-631f-487f-bcd1-a3a594979839\") " pod="openshift-image-registry/image-registry-5cc57669fb-7w5wx" Jul 23 00:12:29.904677 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:29.904652 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1243e526-631f-487f-bcd1-a3a594979839-bound-sa-token\") pod \"image-registry-5cc57669fb-7w5wx\" (UID: \"1243e526-631f-487f-bcd1-a3a594979839\") " pod="openshift-image-registry/image-registry-5cc57669fb-7w5wx" Jul 23 00:12:29.904785 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:29.904708 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5ssj\" (UniqueName: \"kubernetes.io/projected/1243e526-631f-487f-bcd1-a3a594979839-kube-api-access-v5ssj\") pod \"image-registry-5cc57669fb-7w5wx\" (UID: \"1243e526-631f-487f-bcd1-a3a594979839\") " pod="openshift-image-registry/image-registry-5cc57669fb-7w5wx" Jul 23 00:12:30.022467 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:30.022424 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cc57669fb-7w5wx" Jul 23 00:12:30.155615 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:30.155576 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5cc57669fb-7w5wx"] Jul 23 00:12:30.158920 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:12:30.158888 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1243e526_631f_487f_bcd1_a3a594979839.slice/crio-34fd96eea6a32fc3163aa10111d19aa35edc33e21cad6f01ed2ced71d48c7552 WatchSource:0}: Error finding container 34fd96eea6a32fc3163aa10111d19aa35edc33e21cad6f01ed2ced71d48c7552: Status 404 returned error can't find the container with id 34fd96eea6a32fc3163aa10111d19aa35edc33e21cad6f01ed2ced71d48c7552 Jul 23 00:12:30.778692 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:30.778654 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cc57669fb-7w5wx" event={"ID":"1243e526-631f-487f-bcd1-a3a594979839","Type":"ContainerStarted","Data":"b63ccfc252599ac267080f8f80de8ef5ba0ad6e6b63895876d3e24f981125c26"} Jul 23 00:12:30.779100 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:30.778708 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cc57669fb-7w5wx" event={"ID":"1243e526-631f-487f-bcd1-a3a594979839","Type":"ContainerStarted","Data":"34fd96eea6a32fc3163aa10111d19aa35edc33e21cad6f01ed2ced71d48c7552"} Jul 23 00:12:30.779100 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:30.778784 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5cc57669fb-7w5wx" Jul 23 00:12:33.422783 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:33.422715 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0ba4e0d-2b3e-4a20-8f3e-1296830be034-service-ca-bundle\") pod \"router-default-76974999cb-mggnm\" (UID: \"c0ba4e0d-2b3e-4a20-8f3e-1296830be034\") " pod="openshift-ingress/router-default-76974999cb-mggnm" Jul 23 00:12:33.423383 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:33.423360 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0ba4e0d-2b3e-4a20-8f3e-1296830be034-service-ca-bundle\") pod \"router-default-76974999cb-mggnm\" (UID: \"c0ba4e0d-2b3e-4a20-8f3e-1296830be034\") " pod="openshift-ingress/router-default-76974999cb-mggnm" Jul 23 00:12:33.666689 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:33.666656 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-qt5sz\"" Jul 23 00:12:33.675024 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:33.674964 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-76974999cb-mggnm" Jul 23 00:12:33.812177 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:33.812119 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5cc57669fb-7w5wx" podStartSLOduration=4.812100389 podStartE2EDuration="4.812100389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:12:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:12:30.798911975 +0000 UTC m=+264.594418832" watchObservedRunningTime="2026-07-23 00:12:33.812100389 +0000 UTC m=+267.607607245" Jul 23 00:12:33.812431 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:33.812417 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-76974999cb-mggnm"] Jul 23 00:12:33.815824 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:12:33.815790 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0ba4e0d_2b3e_4a20_8f3e_1296830be034.slice/crio-6744da18e4e7e4a399f146b1a167df39fbaf0867e3d3f204f20551db60870984 WatchSource:0}: Error finding container 6744da18e4e7e4a399f146b1a167df39fbaf0867e3d3f204f20551db60870984: Status 404 returned error can't find the container with id 6744da18e4e7e4a399f146b1a167df39fbaf0867e3d3f204f20551db60870984 Jul 23 00:12:34.791746 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:34.791707 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-76974999cb-mggnm" event={"ID":"c0ba4e0d-2b3e-4a20-8f3e-1296830be034","Type":"ContainerStarted","Data":"ec6721bc5145d5a126adbac3bc5a11741635cc3decdcfb7151fb5cf08d988f58"} Jul 23 00:12:34.792180 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:34.791775 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-76974999cb-mggnm" event={"ID":"c0ba4e0d-2b3e-4a20-8f3e-1296830be034","Type":"ContainerStarted","Data":"6744da18e4e7e4a399f146b1a167df39fbaf0867e3d3f204f20551db60870984"} Jul 23 00:12:34.813449 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:34.813393 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-76974999cb-mggnm" podStartSLOduration=33.813375584 podStartE2EDuration="33.813375584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:12:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:12:34.811625427 +0000 UTC m=+268.607132283" watchObservedRunningTime="2026-07-23 00:12:34.813375584 +0000 UTC m=+268.608882440" Jul 23 00:12:35.675540 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:35.675500 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-76974999cb-mggnm" Jul 23 00:12:35.678666 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:35.678640 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-76974999cb-mggnm" Jul 23 00:12:35.795301 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:35.795268 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-76974999cb-mggnm" Jul 23 00:12:35.796528 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:35.796508 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-76974999cb-mggnm" Jul 23 00:12:36.750433 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:36.750393 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/462cefbf-6678-4aa7-96ba-66a0630f1a63-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nz9r9\" (UID: \"462cefbf-6678-4aa7-96ba-66a0630f1a63\") " pod="openshift-insights/insights-runtime-extractor-nz9r9" Jul 23 00:12:36.752746 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:36.752723 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/462cefbf-6678-4aa7-96ba-66a0630f1a63-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nz9r9\" (UID: \"462cefbf-6678-4aa7-96ba-66a0630f1a63\") " pod="openshift-insights/insights-runtime-extractor-nz9r9" Jul 23 00:12:37.025109 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:37.025076 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5btmf\"" Jul 23 00:12:37.032183 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:37.032155 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nz9r9" Jul 23 00:12:37.188270 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:37.188240 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nz9r9"] Jul 23 00:12:37.191672 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:12:37.191644 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod462cefbf_6678_4aa7_96ba_66a0630f1a63.slice/crio-be3fed65b613c5e41a2ef7853420ff2ce548100e40c3accc5b1570ce4724d390 WatchSource:0}: Error finding container be3fed65b613c5e41a2ef7853420ff2ce548100e40c3accc5b1570ce4724d390: Status 404 returned error can't find the container with id be3fed65b613c5e41a2ef7853420ff2ce548100e40c3accc5b1570ce4724d390 Jul 23 00:12:37.802633 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:37.802598 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nz9r9" event={"ID":"462cefbf-6678-4aa7-96ba-66a0630f1a63","Type":"ContainerStarted","Data":"be3fed65b613c5e41a2ef7853420ff2ce548100e40c3accc5b1570ce4724d390"} Jul 23 00:12:42.823708 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:42.823677 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nz9r9" event={"ID":"462cefbf-6678-4aa7-96ba-66a0630f1a63","Type":"ContainerStarted","Data":"1a395dfd7651d469dda0bc706bd49c90548f9f801fde861ca15bc07ddec92f67"} Jul 23 00:12:46.841062 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:46.841022 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nz9r9" event={"ID":"462cefbf-6678-4aa7-96ba-66a0630f1a63","Type":"ContainerStarted","Data":"02f3bd05b8366c81599cefa10c23a023c6c5675a5a92d77b4c24b42dddc8aa5e"} Jul 23 00:12:48.487331 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:12:48.487284 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-ndkl6" podUID="dd67adef-fce4-429b-b01b-8225f9c33ab4" Jul 23 00:12:48.847854 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:48.847816 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ndkl6" Jul 23 00:12:49.853398 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:49.853360 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nz9r9" event={"ID":"462cefbf-6678-4aa7-96ba-66a0630f1a63","Type":"ContainerStarted","Data":"426c0adff4219a47bf27485e554952b99a38bcf478900c3c338cf13636715ad0"} Jul 23 00:12:49.873327 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:49.873269 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-nz9r9" podStartSLOduration=33.723854139 podStartE2EDuration="45.873254549s" totalImagesPullingTime="12.14940041s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:12:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:12:49.872367013 +0000 UTC m=+283.667873870" watchObservedRunningTime="2026-07-23 00:12:49.873254549 +0000 UTC m=+283.668761406" Jul 23 00:12:50.027277 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:50.027231 2562 patch_prober.go:28] interesting pod/image-registry-5cc57669fb-7w5wx container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 23 00:12:50.027447 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:50.027292 2562 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5cc57669fb-7w5wx" podUID="1243e526-631f-487f-bcd1-a3a594979839" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 23 00:12:51.571885 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:51.571845 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dd67adef-fce4-429b-b01b-8225f9c33ab4-metrics-tls\") pod \"dns-default-ndkl6\" (UID: \"dd67adef-fce4-429b-b01b-8225f9c33ab4\") " pod="openshift-dns/dns-default-ndkl6" Jul 23 00:12:51.574075 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:51.574053 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dd67adef-fce4-429b-b01b-8225f9c33ab4-metrics-tls\") pod \"dns-default-ndkl6\" (UID: \"dd67adef-fce4-429b-b01b-8225f9c33ab4\") " pod="openshift-dns/dns-default-ndkl6" Jul 23 00:12:51.786085 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:51.786057 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5cc57669fb-7w5wx" Jul 23 00:12:51.851012 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:51.850931 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fhhqg\"" Jul 23 00:12:51.858434 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:51.858400 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ndkl6" Jul 23 00:12:51.989194 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:51.989167 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ndkl6"] Jul 23 00:12:51.991831 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:12:51.991801 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd67adef_fce4_429b_b01b_8225f9c33ab4.slice/crio-3ec0149ab087000a544636fef059132b8ad4ee337bc4db885abdb05aca693e2b WatchSource:0}: Error finding container 3ec0149ab087000a544636fef059132b8ad4ee337bc4db885abdb05aca693e2b: Status 404 returned error can't find the container with id 3ec0149ab087000a544636fef059132b8ad4ee337bc4db885abdb05aca693e2b Jul 23 00:12:52.864532 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:52.864496 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ndkl6" event={"ID":"dd67adef-fce4-429b-b01b-8225f9c33ab4","Type":"ContainerStarted","Data":"3ec0149ab087000a544636fef059132b8ad4ee337bc4db885abdb05aca693e2b"} Jul 23 00:12:53.871945 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:53.871854 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ndkl6" event={"ID":"dd67adef-fce4-429b-b01b-8225f9c33ab4","Type":"ContainerStarted","Data":"ee3627c8ae349b1c5fb2dd226a69a7b1dd5cb1de38133be0be4ec81323ccff89"} Jul 23 00:12:53.871945 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:53.871914 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ndkl6" event={"ID":"dd67adef-fce4-429b-b01b-8225f9c33ab4","Type":"ContainerStarted","Data":"a28a8c57e55c552cca2086d84d2533d29e717f3c02cdc98bf0bd7f2fe223343b"} Jul 23 00:12:53.872434 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:53.871990 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-ndkl6" Jul 23 00:12:53.890516 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:12:53.890461 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-ndkl6" podStartSLOduration=251.357847122 podStartE2EDuration="4m12.890442805s" totalImagesPullingTime="1.532595683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:08:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:12:53.88972324 +0000 UTC m=+287.685230107" watchObservedRunningTime="2026-07-23 00:12:53.890442805 +0000 UTC m=+287.685949700" Jul 23 00:13:00.002748 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:13:00.002693 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-kfgzz" podUID="5a9bdee1-a493-4223-a7b6-3ae1f5024eb6" Jul 23 00:13:01.496312 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:01.496276 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-dscmh"] Jul 23 00:13:01.499513 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:01.499490 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dscmh" Jul 23 00:13:01.503100 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:01.503075 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rtrwh\"" Jul 23 00:13:01.503220 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:01.503128 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 23 00:13:01.503220 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:01.503138 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 23 00:13:01.503220 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:01.503079 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 23 00:13:01.504110 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:01.504094 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 23 00:13:01.504203 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:01.504106 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 23 00:13:01.504558 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:01.504545 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 23 00:13:01.647517 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:01.647472 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/34ac04c8-8aed-457a-b3f0-5235d1f2c421-metrics-client-ca\") pod \"node-exporter-dscmh\" (UID: \"34ac04c8-8aed-457a-b3f0-5235d1f2c421\") " pod="openshift-monitoring/node-exporter-dscmh" Jul 23 00:13:01.647694 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:01.647524 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/34ac04c8-8aed-457a-b3f0-5235d1f2c421-sys\") pod \"node-exporter-dscmh\" (UID: \"34ac04c8-8aed-457a-b3f0-5235d1f2c421\") " pod="openshift-monitoring/node-exporter-dscmh" Jul 23 00:13:01.647694 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:01.647585 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/34ac04c8-8aed-457a-b3f0-5235d1f2c421-root\") pod \"node-exporter-dscmh\" (UID: \"34ac04c8-8aed-457a-b3f0-5235d1f2c421\") " pod="openshift-monitoring/node-exporter-dscmh" Jul 23 00:13:01.647694 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:01.647607 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/34ac04c8-8aed-457a-b3f0-5235d1f2c421-node-exporter-wtmp\") pod \"node-exporter-dscmh\" (UID: \"34ac04c8-8aed-457a-b3f0-5235d1f2c421\") " pod="openshift-monitoring/node-exporter-dscmh" Jul 23 00:13:01.647694 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:01.647634 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/34ac04c8-8aed-457a-b3f0-5235d1f2c421-node-exporter-accelerators-collector-config\") pod \"node-exporter-dscmh\" (UID: \"34ac04c8-8aed-457a-b3f0-5235d1f2c421\") " pod="openshift-monitoring/node-exporter-dscmh" Jul 23 00:13:01.647694 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:01.647662 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/34ac04c8-8aed-457a-b3f0-5235d1f2c421-node-exporter-tls\") pod \"node-exporter-dscmh\" (UID: \"34ac04c8-8aed-457a-b3f0-5235d1f2c421\") " pod="openshift-monitoring/node-exporter-dscmh" Jul 23 00:13:01.647694 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:01.647677 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/34ac04c8-8aed-457a-b3f0-5235d1f2c421-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dscmh\" (UID: \"34ac04c8-8aed-457a-b3f0-5235d1f2c421\") " pod="openshift-monitoring/node-exporter-dscmh" Jul 23 00:13:01.647961 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:01.647714 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twtfv\" (UniqueName: \"kubernetes.io/projected/34ac04c8-8aed-457a-b3f0-5235d1f2c421-kube-api-access-twtfv\") pod \"node-exporter-dscmh\" (UID: \"34ac04c8-8aed-457a-b3f0-5235d1f2c421\") " pod="openshift-monitoring/node-exporter-dscmh" Jul 23 00:13:01.647961 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:01.647799 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/34ac04c8-8aed-457a-b3f0-5235d1f2c421-node-exporter-textfile\") pod \"node-exporter-dscmh\" (UID: \"34ac04c8-8aed-457a-b3f0-5235d1f2c421\") " pod="openshift-monitoring/node-exporter-dscmh" Jul 23 00:13:01.748656 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:01.748562 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/34ac04c8-8aed-457a-b3f0-5235d1f2c421-sys\") pod \"node-exporter-dscmh\" (UID: \"34ac04c8-8aed-457a-b3f0-5235d1f2c421\") " pod="openshift-monitoring/node-exporter-dscmh" Jul 23 00:13:01.748656 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:01.748612 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/34ac04c8-8aed-457a-b3f0-5235d1f2c421-root\") pod \"node-exporter-dscmh\" (UID: \"34ac04c8-8aed-457a-b3f0-5235d1f2c421\") " pod="openshift-monitoring/node-exporter-dscmh" Jul 23 00:13:01.748656 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:01.748631 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/34ac04c8-8aed-457a-b3f0-5235d1f2c421-node-exporter-wtmp\") pod \"node-exporter-dscmh\" (UID: \"34ac04c8-8aed-457a-b3f0-5235d1f2c421\") " pod="openshift-monitoring/node-exporter-dscmh" Jul 23 00:13:01.748971 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:01.748659 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/34ac04c8-8aed-457a-b3f0-5235d1f2c421-node-exporter-accelerators-collector-config\") pod \"node-exporter-dscmh\" (UID: \"34ac04c8-8aed-457a-b3f0-5235d1f2c421\") " pod="openshift-monitoring/node-exporter-dscmh" Jul 23 00:13:01.748971 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:01.748679 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/34ac04c8-8aed-457a-b3f0-5235d1f2c421-sys\") pod \"node-exporter-dscmh\" (UID: \"34ac04c8-8aed-457a-b3f0-5235d1f2c421\") " pod="openshift-monitoring/node-exporter-dscmh" Jul 23 00:13:01.748971 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:01.748682 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/34ac04c8-8aed-457a-b3f0-5235d1f2c421-root\") pod \"node-exporter-dscmh\" (UID: \"34ac04c8-8aed-457a-b3f0-5235d1f2c421\") " pod="openshift-monitoring/node-exporter-dscmh" Jul 23 00:13:01.748971 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:01.748689 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/34ac04c8-8aed-457a-b3f0-5235d1f2c421-node-exporter-tls\") pod \"node-exporter-dscmh\" (UID: \"34ac04c8-8aed-457a-b3f0-5235d1f2c421\") " pod="openshift-monitoring/node-exporter-dscmh" Jul 23 00:13:01.748971 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:01.748744 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/34ac04c8-8aed-457a-b3f0-5235d1f2c421-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dscmh\" (UID: \"34ac04c8-8aed-457a-b3f0-5235d1f2c421\") " pod="openshift-monitoring/node-exporter-dscmh" Jul 23 00:13:01.748971 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:13:01.748805 2562 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 23 00:13:01.748971 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:01.748810 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/34ac04c8-8aed-457a-b3f0-5235d1f2c421-node-exporter-wtmp\") pod \"node-exporter-dscmh\" (UID: \"34ac04c8-8aed-457a-b3f0-5235d1f2c421\") " pod="openshift-monitoring/node-exporter-dscmh" Jul 23 00:13:01.748971 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:01.748815 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twtfv\" (UniqueName: \"kubernetes.io/projected/34ac04c8-8aed-457a-b3f0-5235d1f2c421-kube-api-access-twtfv\") pod \"node-exporter-dscmh\" (UID: \"34ac04c8-8aed-457a-b3f0-5235d1f2c421\") " pod="openshift-monitoring/node-exporter-dscmh" Jul 23 00:13:01.748971 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:13:01.748875 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34ac04c8-8aed-457a-b3f0-5235d1f2c421-node-exporter-tls podName:34ac04c8-8aed-457a-b3f0-5235d1f2c421 nodeName:}" failed. No retries permitted until 2026-07-23 00:13:02.248855879 +0000 UTC m=+296.044362723 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/34ac04c8-8aed-457a-b3f0-5235d1f2c421-node-exporter-tls") pod "node-exporter-dscmh" (UID: "34ac04c8-8aed-457a-b3f0-5235d1f2c421") : secret "node-exporter-tls" not found Jul 23 00:13:01.748971 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:01.748918 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/34ac04c8-8aed-457a-b3f0-5235d1f2c421-node-exporter-textfile\") pod \"node-exporter-dscmh\" (UID: \"34ac04c8-8aed-457a-b3f0-5235d1f2c421\") " pod="openshift-monitoring/node-exporter-dscmh" Jul 23 00:13:01.748971 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:01.748954 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/34ac04c8-8aed-457a-b3f0-5235d1f2c421-metrics-client-ca\") pod \"node-exporter-dscmh\" (UID: \"34ac04c8-8aed-457a-b3f0-5235d1f2c421\") " pod="openshift-monitoring/node-exporter-dscmh" Jul 23 00:13:01.749459 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:01.749167 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/34ac04c8-8aed-457a-b3f0-5235d1f2c421-node-exporter-textfile\") pod \"node-exporter-dscmh\" (UID: \"34ac04c8-8aed-457a-b3f0-5235d1f2c421\") " pod="openshift-monitoring/node-exporter-dscmh" Jul 23 00:13:01.749459 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:01.749238 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/34ac04c8-8aed-457a-b3f0-5235d1f2c421-node-exporter-accelerators-collector-config\") pod \"node-exporter-dscmh\" (UID: \"34ac04c8-8aed-457a-b3f0-5235d1f2c421\") " pod="openshift-monitoring/node-exporter-dscmh" Jul 23 00:13:01.749459 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:01.749385 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/34ac04c8-8aed-457a-b3f0-5235d1f2c421-metrics-client-ca\") pod \"node-exporter-dscmh\" (UID: \"34ac04c8-8aed-457a-b3f0-5235d1f2c421\") " pod="openshift-monitoring/node-exporter-dscmh" Jul 23 00:13:01.751095 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:01.751073 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/34ac04c8-8aed-457a-b3f0-5235d1f2c421-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dscmh\" (UID: \"34ac04c8-8aed-457a-b3f0-5235d1f2c421\") " pod="openshift-monitoring/node-exporter-dscmh" Jul 23 00:13:01.757805 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:01.757745 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twtfv\" (UniqueName: \"kubernetes.io/projected/34ac04c8-8aed-457a-b3f0-5235d1f2c421-kube-api-access-twtfv\") pod \"node-exporter-dscmh\" (UID: \"34ac04c8-8aed-457a-b3f0-5235d1f2c421\") " pod="openshift-monitoring/node-exporter-dscmh" Jul 23 00:13:02.254329 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:02.254274 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/34ac04c8-8aed-457a-b3f0-5235d1f2c421-node-exporter-tls\") pod \"node-exporter-dscmh\" (UID: \"34ac04c8-8aed-457a-b3f0-5235d1f2c421\") " pod="openshift-monitoring/node-exporter-dscmh" Jul 23 00:13:02.256623 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:02.256598 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/34ac04c8-8aed-457a-b3f0-5235d1f2c421-node-exporter-tls\") pod \"node-exporter-dscmh\" (UID: \"34ac04c8-8aed-457a-b3f0-5235d1f2c421\") " pod="openshift-monitoring/node-exporter-dscmh" Jul 23 00:13:02.408668 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:02.408631 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dscmh" Jul 23 00:13:02.420219 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:13:02.420191 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34ac04c8_8aed_457a_b3f0_5235d1f2c421.slice/crio-5e07f1f783980b25257e19021b0a74e72d9beae74255429dd16cf61dbabe115b WatchSource:0}: Error finding container 5e07f1f783980b25257e19021b0a74e72d9beae74255429dd16cf61dbabe115b: Status 404 returned error can't find the container with id 5e07f1f783980b25257e19021b0a74e72d9beae74255429dd16cf61dbabe115b Jul 23 00:13:02.551812 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:02.551778 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 23 00:13:02.556344 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:02.556323 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:13:02.559315 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:02.559283 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 23 00:13:02.559315 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:02.559291 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 23 00:13:02.559535 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:02.559292 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 23 00:13:02.559535 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:02.559297 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 23 00:13:02.559925 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:02.559872 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 23 00:13:02.559925 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:02.559893 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 23 00:13:02.560089 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:02.559984 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 23 00:13:02.560143 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:02.560120 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 23 00:13:02.560244 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:02.560228 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-jvpp4\"" Jul 23 00:13:02.560322 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:02.560254 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 23 00:13:02.568847 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:02.568817 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 23 00:13:02.658156 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:02.658111 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rxp4\" (UniqueName: \"kubernetes.io/projected/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-kube-api-access-4rxp4\") pod \"alertmanager-main-0\" (UID: \"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:13:02.658360 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:02.658172 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:13:02.658360 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:02.658202 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:13:02.658360 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:02.658224 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-config-out\") pod \"alertmanager-main-0\" (UID: \"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:13:02.658360 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:02.658245 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:13:02.658360 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:02.658338 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:13:02.658360 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:02.658358 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:13:02.658658 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:02.658385 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:13:02.658658 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:02.658411 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-config-volume\") pod \"alertmanager-main-0\" (UID: \"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:13:02.658658 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:02.658437 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:13:02.658658 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:02.658504 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:13:02.658658 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:02.658543 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-web-config\") pod \"alertmanager-main-0\" (UID: \"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:13:02.658658 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:02.658570 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:13:02.759285 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:02.759240 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:13:02.759285 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:02.759289 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:13:02.759554 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:02.759310 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:13:02.759554 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:02.759330 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-config-volume\") pod \"alertmanager-main-0\" (UID: \"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:13:02.759554 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:02.759356 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:13:02.759554 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:02.759388 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:13:02.759554 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:02.759421 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-web-config\") pod \"alertmanager-main-0\" (UID: \"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:13:02.759554 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:13:02.759419 2562 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 23 00:13:02.759554 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:02.759467 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:13:02.759554 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:13:02.759501 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-secret-alertmanager-main-tls podName:bd760290-ebfd-4cdb-9df6-a6ff5e409fe9 nodeName:}" failed. No retries permitted until 2026-07-23 00:13:03.259480268 +0000 UTC m=+297.054987111 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "bd760290-ebfd-4cdb-9df6-a6ff5e409fe9") : secret "alertmanager-main-tls" not found Jul 23 00:13:02.759554 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:02.759544 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rxp4\" (UniqueName: \"kubernetes.io/projected/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-kube-api-access-4rxp4\") pod \"alertmanager-main-0\" (UID: \"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:13:02.760018 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:02.759617 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:13:02.760018 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:02.759643 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:13:02.760018 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:02.759666 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-config-out\") pod \"alertmanager-main-0\" (UID: \"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:13:02.760018 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:02.759690 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:13:02.760669 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:02.760636 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:13:02.761449 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:02.761406 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:13:02.761611 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:02.761437 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:13:02.762711 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:02.762663 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:13:02.762960 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:02.762936 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:13:02.762960 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:02.762945 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-config-out\") pod \"alertmanager-main-0\" (UID: \"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:13:02.763097 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:02.763073 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:13:02.763156 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:02.763095 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-config-volume\") pod \"alertmanager-main-0\" (UID: \"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:13:02.763776 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:02.763739 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:13:02.763840 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:02.763797 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-web-config\") pod \"alertmanager-main-0\" (UID: \"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:13:02.764268 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:02.764234 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:13:02.769100 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:02.769082 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rxp4\" (UniqueName: \"kubernetes.io/projected/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-kube-api-access-4rxp4\") pod \"alertmanager-main-0\" (UID: \"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:13:02.902398 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:02.902298 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dscmh" event={"ID":"34ac04c8-8aed-457a-b3f0-5235d1f2c421","Type":"ContainerStarted","Data":"5e07f1f783980b25257e19021b0a74e72d9beae74255429dd16cf61dbabe115b"} Jul 23 00:13:03.264232 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:03.264141 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:13:03.264403 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:13:03.264332 2562 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 23 00:13:03.264463 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:13:03.264420 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-secret-alertmanager-main-tls podName:bd760290-ebfd-4cdb-9df6-a6ff5e409fe9 nodeName:}" failed. No retries permitted until 2026-07-23 00:13:04.264398091 +0000 UTC m=+298.059904930 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "bd760290-ebfd-4cdb-9df6-a6ff5e409fe9") : secret "alertmanager-main-tls" not found Jul 23 00:13:03.458055 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:03.458013 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-94f4d8999-2kqks"] Jul 23 00:13:03.461750 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:03.461723 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-94f4d8999-2kqks" Jul 23 00:13:03.464953 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:03.464872 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 23 00:13:03.465222 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:03.465185 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 23 00:13:03.465222 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:03.465198 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 23 00:13:03.465222 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:03.465219 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-ftolptesr960t\"" Jul 23 00:13:03.465433 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:03.465198 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-g6vtv\"" Jul 23 00:13:03.465433 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:03.465202 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 23 00:13:03.465433 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:03.465322 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 23 00:13:03.472263 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:03.472231 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-94f4d8999-2kqks"] Jul 23 00:13:03.566114 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:03.566071 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a54ddaab-286b-4d31-ad83-8f59cdb42c9a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-94f4d8999-2kqks\" (UID: \"a54ddaab-286b-4d31-ad83-8f59cdb42c9a\") " pod="openshift-monitoring/thanos-querier-94f4d8999-2kqks" Jul 23 00:13:03.566479 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:03.566133 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a54ddaab-286b-4d31-ad83-8f59cdb42c9a-secret-thanos-querier-tls\") pod \"thanos-querier-94f4d8999-2kqks\" (UID: \"a54ddaab-286b-4d31-ad83-8f59cdb42c9a\") " pod="openshift-monitoring/thanos-querier-94f4d8999-2kqks" Jul 23 00:13:03.566479 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:03.566160 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a54ddaab-286b-4d31-ad83-8f59cdb42c9a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-94f4d8999-2kqks\" (UID: \"a54ddaab-286b-4d31-ad83-8f59cdb42c9a\") " pod="openshift-monitoring/thanos-querier-94f4d8999-2kqks" Jul 23 00:13:03.566479 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:03.566199 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a54ddaab-286b-4d31-ad83-8f59cdb42c9a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-94f4d8999-2kqks\" (UID: \"a54ddaab-286b-4d31-ad83-8f59cdb42c9a\") " pod="openshift-monitoring/thanos-querier-94f4d8999-2kqks" Jul 23 00:13:03.566479 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:03.566229 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r7g5\" (UniqueName: \"kubernetes.io/projected/a54ddaab-286b-4d31-ad83-8f59cdb42c9a-kube-api-access-8r7g5\") pod \"thanos-querier-94f4d8999-2kqks\" (UID: \"a54ddaab-286b-4d31-ad83-8f59cdb42c9a\") " pod="openshift-monitoring/thanos-querier-94f4d8999-2kqks" Jul 23 00:13:03.566479 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:03.566253 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a54ddaab-286b-4d31-ad83-8f59cdb42c9a-secret-grpc-tls\") pod \"thanos-querier-94f4d8999-2kqks\" (UID: \"a54ddaab-286b-4d31-ad83-8f59cdb42c9a\") " pod="openshift-monitoring/thanos-querier-94f4d8999-2kqks" Jul 23 00:13:03.566479 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:03.566324 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a54ddaab-286b-4d31-ad83-8f59cdb42c9a-metrics-client-ca\") pod \"thanos-querier-94f4d8999-2kqks\" (UID: \"a54ddaab-286b-4d31-ad83-8f59cdb42c9a\") " pod="openshift-monitoring/thanos-querier-94f4d8999-2kqks" Jul 23 00:13:03.566479 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:03.566387 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a54ddaab-286b-4d31-ad83-8f59cdb42c9a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-94f4d8999-2kqks\" (UID: \"a54ddaab-286b-4d31-ad83-8f59cdb42c9a\") " pod="openshift-monitoring/thanos-querier-94f4d8999-2kqks" Jul 23 00:13:03.666778 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:03.666714 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a54ddaab-286b-4d31-ad83-8f59cdb42c9a-secret-thanos-querier-tls\") pod \"thanos-querier-94f4d8999-2kqks\" (UID: \"a54ddaab-286b-4d31-ad83-8f59cdb42c9a\") " pod="openshift-monitoring/thanos-querier-94f4d8999-2kqks" Jul 23 00:13:03.666778 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:03.666781 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a54ddaab-286b-4d31-ad83-8f59cdb42c9a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-94f4d8999-2kqks\" (UID: \"a54ddaab-286b-4d31-ad83-8f59cdb42c9a\") " pod="openshift-monitoring/thanos-querier-94f4d8999-2kqks" Jul 23 00:13:03.667028 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:03.666809 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a54ddaab-286b-4d31-ad83-8f59cdb42c9a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-94f4d8999-2kqks\" (UID: \"a54ddaab-286b-4d31-ad83-8f59cdb42c9a\") " pod="openshift-monitoring/thanos-querier-94f4d8999-2kqks" Jul 23 00:13:03.667028 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:03.666831 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r7g5\" (UniqueName: \"kubernetes.io/projected/a54ddaab-286b-4d31-ad83-8f59cdb42c9a-kube-api-access-8r7g5\") pod \"thanos-querier-94f4d8999-2kqks\" (UID: \"a54ddaab-286b-4d31-ad83-8f59cdb42c9a\") " pod="openshift-monitoring/thanos-querier-94f4d8999-2kqks" Jul 23 00:13:03.667028 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:03.666849 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a54ddaab-286b-4d31-ad83-8f59cdb42c9a-secret-grpc-tls\") pod \"thanos-querier-94f4d8999-2kqks\" (UID: \"a54ddaab-286b-4d31-ad83-8f59cdb42c9a\") " pod="openshift-monitoring/thanos-querier-94f4d8999-2kqks" Jul 23 00:13:03.667028 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:03.666979 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a54ddaab-286b-4d31-ad83-8f59cdb42c9a-metrics-client-ca\") pod \"thanos-querier-94f4d8999-2kqks\" (UID: \"a54ddaab-286b-4d31-ad83-8f59cdb42c9a\") " pod="openshift-monitoring/thanos-querier-94f4d8999-2kqks" Jul 23 00:13:03.667227 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:03.667054 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a54ddaab-286b-4d31-ad83-8f59cdb42c9a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-94f4d8999-2kqks\" (UID: \"a54ddaab-286b-4d31-ad83-8f59cdb42c9a\") " pod="openshift-monitoring/thanos-querier-94f4d8999-2kqks" Jul 23 00:13:03.667227 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:03.667122 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a54ddaab-286b-4d31-ad83-8f59cdb42c9a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-94f4d8999-2kqks\" (UID: \"a54ddaab-286b-4d31-ad83-8f59cdb42c9a\") " pod="openshift-monitoring/thanos-querier-94f4d8999-2kqks" Jul 23 00:13:03.667851 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:03.667739 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a54ddaab-286b-4d31-ad83-8f59cdb42c9a-metrics-client-ca\") pod \"thanos-querier-94f4d8999-2kqks\" (UID: \"a54ddaab-286b-4d31-ad83-8f59cdb42c9a\") " pod="openshift-monitoring/thanos-querier-94f4d8999-2kqks" Jul 23 00:13:03.669542 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:03.669501 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a54ddaab-286b-4d31-ad83-8f59cdb42c9a-secret-thanos-querier-tls\") pod \"thanos-querier-94f4d8999-2kqks\" (UID: \"a54ddaab-286b-4d31-ad83-8f59cdb42c9a\") " pod="openshift-monitoring/thanos-querier-94f4d8999-2kqks" Jul 23 00:13:03.669542 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:03.669483 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a54ddaab-286b-4d31-ad83-8f59cdb42c9a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-94f4d8999-2kqks\" (UID: \"a54ddaab-286b-4d31-ad83-8f59cdb42c9a\") " pod="openshift-monitoring/thanos-querier-94f4d8999-2kqks" Jul 23 00:13:03.669803 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:03.669781 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a54ddaab-286b-4d31-ad83-8f59cdb42c9a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-94f4d8999-2kqks\" (UID: \"a54ddaab-286b-4d31-ad83-8f59cdb42c9a\") " pod="openshift-monitoring/thanos-querier-94f4d8999-2kqks" Jul 23 00:13:03.669860 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:03.669806 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a54ddaab-286b-4d31-ad83-8f59cdb42c9a-secret-grpc-tls\") pod \"thanos-querier-94f4d8999-2kqks\" (UID: \"a54ddaab-286b-4d31-ad83-8f59cdb42c9a\") " pod="openshift-monitoring/thanos-querier-94f4d8999-2kqks" Jul 23 00:13:03.669989 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:03.669969 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a54ddaab-286b-4d31-ad83-8f59cdb42c9a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-94f4d8999-2kqks\" (UID: \"a54ddaab-286b-4d31-ad83-8f59cdb42c9a\") " pod="openshift-monitoring/thanos-querier-94f4d8999-2kqks" Jul 23 00:13:03.670031 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:03.670002 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a54ddaab-286b-4d31-ad83-8f59cdb42c9a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-94f4d8999-2kqks\" (UID: \"a54ddaab-286b-4d31-ad83-8f59cdb42c9a\") " pod="openshift-monitoring/thanos-querier-94f4d8999-2kqks" Jul 23 00:13:03.675942 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:03.675913 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r7g5\" (UniqueName: \"kubernetes.io/projected/a54ddaab-286b-4d31-ad83-8f59cdb42c9a-kube-api-access-8r7g5\") pod \"thanos-querier-94f4d8999-2kqks\" (UID: \"a54ddaab-286b-4d31-ad83-8f59cdb42c9a\") " pod="openshift-monitoring/thanos-querier-94f4d8999-2kqks" Jul 23 00:13:03.772791 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:03.772743 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-94f4d8999-2kqks" Jul 23 00:13:03.877600 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:03.877569 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-ndkl6" Jul 23 00:13:03.908944 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:03.908903 2562 generic.go:356] "Generic (PLEG): container finished" podID="34ac04c8-8aed-457a-b3f0-5235d1f2c421" containerID="e460c570c62fabd41da0cc7d0ceb997b76114af81bd8257278fa5283341cf84a" exitCode=0 Jul 23 00:13:03.909112 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:03.908955 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dscmh" event={"ID":"34ac04c8-8aed-457a-b3f0-5235d1f2c421","Type":"ContainerDied","Data":"e460c570c62fabd41da0cc7d0ceb997b76114af81bd8257278fa5283341cf84a"} Jul 23 00:13:03.916832 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:03.916796 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-94f4d8999-2kqks"] Jul 23 00:13:03.918137 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:13:03.918107 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda54ddaab_286b_4d31_ad83_8f59cdb42c9a.slice/crio-3bfad60f93b018b3fdc3305d2d805ea4d1d0c221d7936588eceea444edd64149 WatchSource:0}: Error finding container 3bfad60f93b018b3fdc3305d2d805ea4d1d0c221d7936588eceea444edd64149: Status 404 returned error can't find the container with id 3bfad60f93b018b3fdc3305d2d805ea4d1d0c221d7936588eceea444edd64149 Jul 23 00:13:04.272493 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:04.272460 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:13:04.274978 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:04.274926 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:13:04.366700 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:04.366658 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:13:04.545466 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:04.545435 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 23 00:13:04.546386 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:13:04.546358 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd760290_ebfd_4cdb_9df6_a6ff5e409fe9.slice/crio-854a0bd686a12bc8e62a73daa7d557877a5e66ebf08ae69d47d459e2d878c1b7 WatchSource:0}: Error finding container 854a0bd686a12bc8e62a73daa7d557877a5e66ebf08ae69d47d459e2d878c1b7: Status 404 returned error can't find the container with id 854a0bd686a12bc8e62a73daa7d557877a5e66ebf08ae69d47d459e2d878c1b7 Jul 23 00:13:04.915592 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:04.915464 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dscmh" event={"ID":"34ac04c8-8aed-457a-b3f0-5235d1f2c421","Type":"ContainerStarted","Data":"d3913b854c6411a627cd4a9950d1fc9398c3cf85208b8b33d4bad5474582f512"} Jul 23 00:13:04.915592 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:04.915528 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dscmh" event={"ID":"34ac04c8-8aed-457a-b3f0-5235d1f2c421","Type":"ContainerStarted","Data":"6ef3891f3b4d741dad3ee7fc31a35ec3ddcd857872715c5c4fd319835c6b071d"} Jul 23 00:13:04.916804 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:04.916752 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9","Type":"ContainerStarted","Data":"854a0bd686a12bc8e62a73daa7d557877a5e66ebf08ae69d47d459e2d878c1b7"} Jul 23 00:13:04.920393 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:04.920308 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-94f4d8999-2kqks" event={"ID":"a54ddaab-286b-4d31-ad83-8f59cdb42c9a","Type":"ContainerStarted","Data":"3bfad60f93b018b3fdc3305d2d805ea4d1d0c221d7936588eceea444edd64149"} Jul 23 00:13:04.935876 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:04.935806 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-dscmh" podStartSLOduration=2.919332507 podStartE2EDuration="3.935782641s" totalImagesPullingTime="946.526618ms" totalInitContainerRuntime="69.923516ms" isStatefulPod=true podCreationTimestamp="2026-07-23 00:13:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:13:04.935174121 +0000 UTC m=+298.730680990" watchObservedRunningTime="2026-07-23 00:13:04.935782641 +0000 UTC m=+298.731289494" Jul 23 00:13:05.791083 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:05.789635 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-9947f897b-pt65r"] Jul 23 00:13:05.794904 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:05.794876 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-9947f897b-pt65r" Jul 23 00:13:05.800113 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:05.800077 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 23 00:13:05.800278 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:05.800153 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 23 00:13:05.800398 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:05.800380 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 23 00:13:05.800630 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:05.800614 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 23 00:13:05.800873 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:05.800614 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-vmt2g\"" Jul 23 00:13:05.800986 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:05.800662 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-7vtth0e8vaahl\"" Jul 23 00:13:05.802406 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:05.802343 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-9947f897b-pt65r"] Jul 23 00:13:05.889974 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:05.889929 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b64b4a7-4872-4de0-ae43-039812f2cb1f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-9947f897b-pt65r\" (UID: \"4b64b4a7-4872-4de0-ae43-039812f2cb1f\") " pod="openshift-monitoring/metrics-server-9947f897b-pt65r" Jul 23 00:13:05.890181 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:05.889991 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/4b64b4a7-4872-4de0-ae43-039812f2cb1f-secret-metrics-server-client-certs\") pod \"metrics-server-9947f897b-pt65r\" (UID: \"4b64b4a7-4872-4de0-ae43-039812f2cb1f\") " pod="openshift-monitoring/metrics-server-9947f897b-pt65r" Jul 23 00:13:05.890181 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:05.890037 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/4b64b4a7-4872-4de0-ae43-039812f2cb1f-audit-log\") pod \"metrics-server-9947f897b-pt65r\" (UID: \"4b64b4a7-4872-4de0-ae43-039812f2cb1f\") " pod="openshift-monitoring/metrics-server-9947f897b-pt65r" Jul 23 00:13:05.890181 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:05.890063 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxr4t\" (UniqueName: \"kubernetes.io/projected/4b64b4a7-4872-4de0-ae43-039812f2cb1f-kube-api-access-jxr4t\") pod \"metrics-server-9947f897b-pt65r\" (UID: \"4b64b4a7-4872-4de0-ae43-039812f2cb1f\") " pod="openshift-monitoring/metrics-server-9947f897b-pt65r" Jul 23 00:13:05.890181 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:05.890099 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4b64b4a7-4872-4de0-ae43-039812f2cb1f-client-ca-bundle\") pod \"metrics-server-9947f897b-pt65r\" (UID: \"4b64b4a7-4872-4de0-ae43-039812f2cb1f\") " pod="openshift-monitoring/metrics-server-9947f897b-pt65r" Jul 23 00:13:05.890393 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:05.890229 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/4b64b4a7-4872-4de0-ae43-039812f2cb1f-metrics-server-audit-profiles\") pod \"metrics-server-9947f897b-pt65r\" (UID: \"4b64b4a7-4872-4de0-ae43-039812f2cb1f\") " pod="openshift-monitoring/metrics-server-9947f897b-pt65r" Jul 23 00:13:05.890393 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:05.890278 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/4b64b4a7-4872-4de0-ae43-039812f2cb1f-secret-metrics-server-tls\") pod \"metrics-server-9947f897b-pt65r\" (UID: \"4b64b4a7-4872-4de0-ae43-039812f2cb1f\") " pod="openshift-monitoring/metrics-server-9947f897b-pt65r" Jul 23 00:13:05.991724 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:05.991687 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/4b64b4a7-4872-4de0-ae43-039812f2cb1f-audit-log\") pod \"metrics-server-9947f897b-pt65r\" (UID: \"4b64b4a7-4872-4de0-ae43-039812f2cb1f\") " pod="openshift-monitoring/metrics-server-9947f897b-pt65r" Jul 23 00:13:05.992204 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:05.991726 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxr4t\" (UniqueName: \"kubernetes.io/projected/4b64b4a7-4872-4de0-ae43-039812f2cb1f-kube-api-access-jxr4t\") pod \"metrics-server-9947f897b-pt65r\" (UID: \"4b64b4a7-4872-4de0-ae43-039812f2cb1f\") " pod="openshift-monitoring/metrics-server-9947f897b-pt65r" Jul 23 00:13:05.992204 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:05.991789 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4b64b4a7-4872-4de0-ae43-039812f2cb1f-client-ca-bundle\") pod \"metrics-server-9947f897b-pt65r\" (UID: \"4b64b4a7-4872-4de0-ae43-039812f2cb1f\") " pod="openshift-monitoring/metrics-server-9947f897b-pt65r" Jul 23 00:13:05.992204 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:05.991871 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/4b64b4a7-4872-4de0-ae43-039812f2cb1f-metrics-server-audit-profiles\") pod \"metrics-server-9947f897b-pt65r\" (UID: \"4b64b4a7-4872-4de0-ae43-039812f2cb1f\") " pod="openshift-monitoring/metrics-server-9947f897b-pt65r" Jul 23 00:13:05.992204 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:05.991899 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/4b64b4a7-4872-4de0-ae43-039812f2cb1f-secret-metrics-server-tls\") pod \"metrics-server-9947f897b-pt65r\" (UID: \"4b64b4a7-4872-4de0-ae43-039812f2cb1f\") " pod="openshift-monitoring/metrics-server-9947f897b-pt65r" Jul 23 00:13:05.992204 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:05.991929 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b64b4a7-4872-4de0-ae43-039812f2cb1f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-9947f897b-pt65r\" (UID: \"4b64b4a7-4872-4de0-ae43-039812f2cb1f\") " pod="openshift-monitoring/metrics-server-9947f897b-pt65r" Jul 23 00:13:05.992204 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:05.991958 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/4b64b4a7-4872-4de0-ae43-039812f2cb1f-secret-metrics-server-client-certs\") pod \"metrics-server-9947f897b-pt65r\" (UID: \"4b64b4a7-4872-4de0-ae43-039812f2cb1f\") " pod="openshift-monitoring/metrics-server-9947f897b-pt65r" Jul 23 00:13:05.992204 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:05.992163 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/4b64b4a7-4872-4de0-ae43-039812f2cb1f-audit-log\") pod \"metrics-server-9947f897b-pt65r\" (UID: \"4b64b4a7-4872-4de0-ae43-039812f2cb1f\") " pod="openshift-monitoring/metrics-server-9947f897b-pt65r" Jul 23 00:13:05.992822 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:05.992797 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b64b4a7-4872-4de0-ae43-039812f2cb1f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-9947f897b-pt65r\" (UID: \"4b64b4a7-4872-4de0-ae43-039812f2cb1f\") " pod="openshift-monitoring/metrics-server-9947f897b-pt65r" Jul 23 00:13:05.993022 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:05.992988 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/4b64b4a7-4872-4de0-ae43-039812f2cb1f-metrics-server-audit-profiles\") pod \"metrics-server-9947f897b-pt65r\" (UID: \"4b64b4a7-4872-4de0-ae43-039812f2cb1f\") " pod="openshift-monitoring/metrics-server-9947f897b-pt65r" Jul 23 00:13:05.994779 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:05.994719 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4b64b4a7-4872-4de0-ae43-039812f2cb1f-client-ca-bundle\") pod \"metrics-server-9947f897b-pt65r\" (UID: \"4b64b4a7-4872-4de0-ae43-039812f2cb1f\") " pod="openshift-monitoring/metrics-server-9947f897b-pt65r" Jul 23 00:13:05.994959 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:05.994917 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/4b64b4a7-4872-4de0-ae43-039812f2cb1f-secret-metrics-server-tls\") pod \"metrics-server-9947f897b-pt65r\" (UID: \"4b64b4a7-4872-4de0-ae43-039812f2cb1f\") " pod="openshift-monitoring/metrics-server-9947f897b-pt65r" Jul 23 00:13:05.995024 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:05.995002 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/4b64b4a7-4872-4de0-ae43-039812f2cb1f-secret-metrics-server-client-certs\") pod \"metrics-server-9947f897b-pt65r\" (UID: \"4b64b4a7-4872-4de0-ae43-039812f2cb1f\") " pod="openshift-monitoring/metrics-server-9947f897b-pt65r" Jul 23 00:13:06.000821 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:06.000784 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxr4t\" (UniqueName: \"kubernetes.io/projected/4b64b4a7-4872-4de0-ae43-039812f2cb1f-kube-api-access-jxr4t\") pod \"metrics-server-9947f897b-pt65r\" (UID: \"4b64b4a7-4872-4de0-ae43-039812f2cb1f\") " pod="openshift-monitoring/metrics-server-9947f897b-pt65r" Jul 23 00:13:06.107535 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:06.107448 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-9947f897b-pt65r" Jul 23 00:13:06.336958 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:06.336917 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-q6j98"] Jul 23 00:13:06.340710 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:06.340659 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-q6j98" Jul 23 00:13:06.344092 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:06.343937 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 23 00:13:06.344092 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:06.344042 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-bw84c\"" Jul 23 00:13:06.352528 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:06.352493 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-q6j98"] Jul 23 00:13:06.391336 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:06.391304 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-9947f897b-pt65r"] Jul 23 00:13:06.393349 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:13:06.393322 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b64b4a7_4872_4de0_ae43_039812f2cb1f.slice/crio-32d27dfeb52cd5d1e58b3e47e97e035f78555eaecb82f604b25506699a922ba0 WatchSource:0}: Error finding container 32d27dfeb52cd5d1e58b3e47e97e035f78555eaecb82f604b25506699a922ba0: Status 404 returned error can't find the container with id 32d27dfeb52cd5d1e58b3e47e97e035f78555eaecb82f604b25506699a922ba0 Jul 23 00:13:06.496973 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:06.496928 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-852c8\" (UniqueName: \"kubernetes.io/projected/cc3da55a-09e5-4b6b-b2b6-b973078b728a-kube-api-access-852c8\") pod \"monitoring-plugin-7b7b7bc8bf-q6j98\" (UID: \"cc3da55a-09e5-4b6b-b2b6-b973078b728a\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-q6j98" Jul 23 00:13:06.497159 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:06.496990 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cc3da55a-09e5-4b6b-b2b6-b973078b728a-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-q6j98\" (UID: \"cc3da55a-09e5-4b6b-b2b6-b973078b728a\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-q6j98" Jul 23 00:13:06.597656 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:06.597611 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-852c8\" (UniqueName: \"kubernetes.io/projected/cc3da55a-09e5-4b6b-b2b6-b973078b728a-kube-api-access-852c8\") pod \"monitoring-plugin-7b7b7bc8bf-q6j98\" (UID: \"cc3da55a-09e5-4b6b-b2b6-b973078b728a\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-q6j98" Jul 23 00:13:06.597656 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:06.597657 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cc3da55a-09e5-4b6b-b2b6-b973078b728a-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-q6j98\" (UID: \"cc3da55a-09e5-4b6b-b2b6-b973078b728a\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-q6j98" Jul 23 00:13:06.600235 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:06.600198 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cc3da55a-09e5-4b6b-b2b6-b973078b728a-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-q6j98\" (UID: \"cc3da55a-09e5-4b6b-b2b6-b973078b728a\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-q6j98" Jul 23 00:13:06.606015 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:06.605995 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-852c8\" (UniqueName: \"kubernetes.io/projected/cc3da55a-09e5-4b6b-b2b6-b973078b728a-kube-api-access-852c8\") pod \"monitoring-plugin-7b7b7bc8bf-q6j98\" (UID: \"cc3da55a-09e5-4b6b-b2b6-b973078b728a\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-q6j98" Jul 23 00:13:06.699247 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:06.699157 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-bw84c\"" Jul 23 00:13:06.706988 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:06.706958 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-q6j98" Jul 23 00:13:06.763072 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:06.763048 2562 kubelet.go:1648] "Image garbage collection succeeded" Jul 23 00:13:06.850855 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:06.850820 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-q6j98"] Jul 23 00:13:06.854459 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:13:06.854425 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc3da55a_09e5_4b6b_b2b6_b973078b728a.slice/crio-8eb9f449a0f1f77a821720fb98e8f8153da12e7d6bf65799cd7499ff34273c38 WatchSource:0}: Error finding container 8eb9f449a0f1f77a821720fb98e8f8153da12e7d6bf65799cd7499ff34273c38: Status 404 returned error can't find the container with id 8eb9f449a0f1f77a821720fb98e8f8153da12e7d6bf65799cd7499ff34273c38 Jul 23 00:13:06.930438 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:06.930394 2562 generic.go:356] "Generic (PLEG): container finished" podID="bd760290-ebfd-4cdb-9df6-a6ff5e409fe9" containerID="8b6345a2805d2a3717621710c802a01d7e4b1234e4a8655b9497331c04b90104" exitCode=0 Jul 23 00:13:06.930603 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:06.930555 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9","Type":"ContainerDied","Data":"8b6345a2805d2a3717621710c802a01d7e4b1234e4a8655b9497331c04b90104"} Jul 23 00:13:06.933638 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:06.933607 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-9947f897b-pt65r" event={"ID":"4b64b4a7-4872-4de0-ae43-039812f2cb1f","Type":"ContainerStarted","Data":"32d27dfeb52cd5d1e58b3e47e97e035f78555eaecb82f604b25506699a922ba0"} Jul 23 00:13:06.936378 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:06.936327 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-94f4d8999-2kqks" event={"ID":"a54ddaab-286b-4d31-ad83-8f59cdb42c9a","Type":"ContainerStarted","Data":"d646a2bcc4854160f9a69cec0943977d27559257a6eebbc5b7dd8579e33dc86c"} Jul 23 00:13:06.936499 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:06.936396 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-94f4d8999-2kqks" event={"ID":"a54ddaab-286b-4d31-ad83-8f59cdb42c9a","Type":"ContainerStarted","Data":"716fe14c0f95f7db2018e27defba747508ee7ca6b208047a78ff029f55c55242"} Jul 23 00:13:06.936499 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:06.936438 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-94f4d8999-2kqks" event={"ID":"a54ddaab-286b-4d31-ad83-8f59cdb42c9a","Type":"ContainerStarted","Data":"ccaedfa1813eff17a44ced38a57a36065a8eb7dd1d89948e3ea5c297a7688eeb"} Jul 23 00:13:06.937681 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:06.937661 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-q6j98" event={"ID":"cc3da55a-09e5-4b6b-b2b6-b973078b728a","Type":"ContainerStarted","Data":"8eb9f449a0f1f77a821720fb98e8f8153da12e7d6bf65799cd7499ff34273c38"} Jul 23 00:13:08.950031 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:08.949980 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-9947f897b-pt65r" event={"ID":"4b64b4a7-4872-4de0-ae43-039812f2cb1f","Type":"ContainerStarted","Data":"0bd2639b78064ac6cacc705df773d9bbfc84878a5f727b79ced1d4e572500a75"} Jul 23 00:13:08.953342 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:08.953220 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-94f4d8999-2kqks" event={"ID":"a54ddaab-286b-4d31-ad83-8f59cdb42c9a","Type":"ContainerStarted","Data":"d962423028df464d22c052874fd0645fc880b92ebafdd4ee39f396f74dec26b7"} Jul 23 00:13:08.953342 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:08.953287 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-94f4d8999-2kqks" event={"ID":"a54ddaab-286b-4d31-ad83-8f59cdb42c9a","Type":"ContainerStarted","Data":"23ea5b44c51ac94e23b8cf2f1ecb92f32764997ff3e8b6b82776d34d648510e1"} Jul 23 00:13:08.953342 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:08.953314 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-94f4d8999-2kqks" event={"ID":"a54ddaab-286b-4d31-ad83-8f59cdb42c9a","Type":"ContainerStarted","Data":"22a39b539643d2e0e8030bea849f5c5fb36fbd444476528b282af252dcc312f3"} Jul 23 00:13:08.953743 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:08.953359 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-94f4d8999-2kqks" Jul 23 00:13:08.968215 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:08.968157 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-9947f897b-pt65r" podStartSLOduration=2.461313264 podStartE2EDuration="3.968139296s" totalImagesPullingTime="1.506826032s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:13:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:13:08.967542754 +0000 UTC m=+302.763049614" watchObservedRunningTime="2026-07-23 00:13:08.968139296 +0000 UTC m=+302.763646152" Jul 23 00:13:08.989833 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:08.989772 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-94f4d8999-2kqks" podStartSLOduration=2.272192769 podStartE2EDuration="5.989739352s" totalImagesPullingTime="3.717546583s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:13:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:13:08.988129371 +0000 UTC m=+302.783636219" watchObservedRunningTime="2026-07-23 00:13:08.989739352 +0000 UTC m=+302.785246209" Jul 23 00:13:09.958532 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:09.958494 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-q6j98" event={"ID":"cc3da55a-09e5-4b6b-b2b6-b973078b728a","Type":"ContainerStarted","Data":"b5545dbcc13a7d9b99d2d78d05a507e4d63f8d5b80e4eb9ad34895406a7f0dbe"} Jul 23 00:13:09.959001 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:09.958649 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-q6j98" Jul 23 00:13:09.962370 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:09.962311 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9","Type":"ContainerStarted","Data":"1ceeda90e86c36df6ba2076a315a4cfba4c20fca9a547843260ca82bb556d260"} Jul 23 00:13:09.962536 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:09.962398 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9","Type":"ContainerStarted","Data":"3e67696fdacf467fd1361eef450ae034675ae2ce102d470dbc9ea131ec4b9cb5"} Jul 23 00:13:09.962536 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:09.962481 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9","Type":"ContainerStarted","Data":"6c1751ed08f9c10d74456a5cee6ee9c413952984c4aafa1f5915a44a5d5559c6"} Jul 23 00:13:09.962536 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:09.962524 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9","Type":"ContainerStarted","Data":"145b0ae1736093328914513480b6c44ff7e7b9c015a2360aaffec97896244fbd"} Jul 23 00:13:09.962666 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:09.962554 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9","Type":"ContainerStarted","Data":"a1a82725bb2b176dfb9ddda01fd36435b0f40c912971e1a3612cf7ddf2c48407"} Jul 23 00:13:09.962666 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:09.962577 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9","Type":"ContainerStarted","Data":"3dd3144834eecb1afad31e508d0d658cfd70859f028f6367252b9f75e17eb6d7"} Jul 23 00:13:09.964469 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:09.964443 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-q6j98" Jul 23 00:13:09.977966 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:09.977915 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-q6j98" podStartSLOduration=1.346436966 podStartE2EDuration="3.977899027s" totalImagesPullingTime="2.631462061s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:13:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:13:09.976844229 +0000 UTC m=+303.772351087" watchObservedRunningTime="2026-07-23 00:13:09.977899027 +0000 UTC m=+303.773405884" Jul 23 00:13:10.004878 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:10.004770 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.755906519 podStartE2EDuration="8.004737887s" totalImagesPullingTime="4.222371216s" totalInitContainerRuntime="26.460152ms" isStatefulPod=false podCreationTimestamp="2026-07-23 00:13:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:13:10.004687481 +0000 UTC m=+303.800194350" watchObservedRunningTime="2026-07-23 00:13:10.004737887 +0000 UTC m=+303.800244744" Jul 23 00:13:11.001630 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:11.001592 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfgzz" Jul 23 00:13:14.968352 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:14.968322 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-94f4d8999-2kqks" Jul 23 00:13:20.002495 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:20.002461 2562 generic.go:356] "Generic (PLEG): container finished" podID="ec2a61a7-0a82-4f9a-aac7-001dec5f5a6f" containerID="593b715fa2847d8812e2fc16736414eb6eff8f974fbf416ab66e50c53678ad8d" exitCode=0 Jul 23 00:13:20.003091 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:20.002529 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-w24mg" event={"ID":"ec2a61a7-0a82-4f9a-aac7-001dec5f5a6f","Type":"ContainerDied","Data":"593b715fa2847d8812e2fc16736414eb6eff8f974fbf416ab66e50c53678ad8d"} Jul 23 00:13:20.003091 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:20.002856 2562 scope.go:117] "RemoveContainer" containerID="593b715fa2847d8812e2fc16736414eb6eff8f974fbf416ab66e50c53678ad8d" Jul 23 00:13:21.008441 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:21.008404 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-w24mg" event={"ID":"ec2a61a7-0a82-4f9a-aac7-001dec5f5a6f","Type":"ContainerStarted","Data":"00dd8832a1db2349c5426769e1bc61573bdbf1336fac5b400ddc06974530092f"} Jul 23 00:13:26.108428 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:26.108388 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-9947f897b-pt65r" Jul 23 00:13:26.108853 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:26.108697 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-9947f897b-pt65r" Jul 23 00:13:36.073241 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:36.073207 2562 generic.go:356] "Generic (PLEG): container finished" podID="190fa13e-db15-4ace-91ed-7d1fd190fe6c" containerID="87440c9e4fff04803e43bde32227f394d9bbe62c8028d1266909c9fdb83fc975" exitCode=0 Jul 23 00:13:36.073730 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:36.073274 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-pkvw6" event={"ID":"190fa13e-db15-4ace-91ed-7d1fd190fe6c","Type":"ContainerDied","Data":"87440c9e4fff04803e43bde32227f394d9bbe62c8028d1266909c9fdb83fc975"} Jul 23 00:13:36.073730 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:36.073632 2562 scope.go:117] "RemoveContainer" containerID="87440c9e4fff04803e43bde32227f394d9bbe62c8028d1266909c9fdb83fc975" Jul 23 00:13:37.078610 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:37.078572 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-pkvw6" event={"ID":"190fa13e-db15-4ace-91ed-7d1fd190fe6c","Type":"ContainerStarted","Data":"594aeab37bbb5f8167c43660a7f8a0319aec4d5573562c0d0b42abc04f6ef192"} Jul 23 00:13:46.113813 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:46.113779 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-9947f897b-pt65r" Jul 23 00:13:46.117699 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:13:46.117673 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-9947f897b-pt65r" Jul 23 00:14:20.623212 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:20.623160 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a9bdee1-a493-4223-a7b6-3ae1f5024eb6-metrics-certs\") pod \"network-metrics-daemon-kfgzz\" (UID: \"5a9bdee1-a493-4223-a7b6-3ae1f5024eb6\") " pod="openshift-multus/network-metrics-daemon-kfgzz" Jul 23 00:14:20.625577 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:20.625552 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a9bdee1-a493-4223-a7b6-3ae1f5024eb6-metrics-certs\") pod \"network-metrics-daemon-kfgzz\" (UID: \"5a9bdee1-a493-4223-a7b6-3ae1f5024eb6\") " pod="openshift-multus/network-metrics-daemon-kfgzz" Jul 23 00:14:20.906640 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:20.906547 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pff25\"" Jul 23 00:14:20.913324 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:20.913296 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfgzz" Jul 23 00:14:21.054093 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:21.054053 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kfgzz"] Jul 23 00:14:21.057010 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:14:21.056972 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a9bdee1_a493_4223_a7b6_3ae1f5024eb6.slice/crio-ed652ed42a7701ffef75a817bdfe78ac6a4758a6e9587071b8fe4b6c5d97ca6c WatchSource:0}: Error finding container ed652ed42a7701ffef75a817bdfe78ac6a4758a6e9587071b8fe4b6c5d97ca6c: Status 404 returned error can't find the container with id ed652ed42a7701ffef75a817bdfe78ac6a4758a6e9587071b8fe4b6c5d97ca6c Jul 23 00:14:21.059402 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:21.059383 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 00:14:21.249158 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:21.249072 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kfgzz" event={"ID":"5a9bdee1-a493-4223-a7b6-3ae1f5024eb6","Type":"ContainerStarted","Data":"ed652ed42a7701ffef75a817bdfe78ac6a4758a6e9587071b8fe4b6c5d97ca6c"} Jul 23 00:14:21.777744 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:21.777703 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 23 00:14:21.778182 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:21.778130 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="bd760290-ebfd-4cdb-9df6-a6ff5e409fe9" containerName="alertmanager" containerID="cri-o://3dd3144834eecb1afad31e508d0d658cfd70859f028f6367252b9f75e17eb6d7" gracePeriod=120 Jul 23 00:14:21.778251 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:21.778195 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="bd760290-ebfd-4cdb-9df6-a6ff5e409fe9" containerName="kube-rbac-proxy-web" containerID="cri-o://145b0ae1736093328914513480b6c44ff7e7b9c015a2360aaffec97896244fbd" gracePeriod=120 Jul 23 00:14:21.778251 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:21.778223 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="bd760290-ebfd-4cdb-9df6-a6ff5e409fe9" containerName="prom-label-proxy" containerID="cri-o://1ceeda90e86c36df6ba2076a315a4cfba4c20fca9a547843260ca82bb556d260" gracePeriod=120 Jul 23 00:14:21.778251 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:21.778229 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="bd760290-ebfd-4cdb-9df6-a6ff5e409fe9" containerName="config-reloader" containerID="cri-o://a1a82725bb2b176dfb9ddda01fd36435b0f40c912971e1a3612cf7ddf2c48407" gracePeriod=120 Jul 23 00:14:21.778406 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:21.778225 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="bd760290-ebfd-4cdb-9df6-a6ff5e409fe9" containerName="kube-rbac-proxy" containerID="cri-o://6c1751ed08f9c10d74456a5cee6ee9c413952984c4aafa1f5915a44a5d5559c6" gracePeriod=120 Jul 23 00:14:21.778406 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:21.778196 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="bd760290-ebfd-4cdb-9df6-a6ff5e409fe9" containerName="kube-rbac-proxy-metric" containerID="cri-o://3e67696fdacf467fd1361eef450ae034675ae2ce102d470dbc9ea131ec4b9cb5" gracePeriod=120 Jul 23 00:14:22.256270 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:22.256226 2562 generic.go:356] "Generic (PLEG): container finished" podID="bd760290-ebfd-4cdb-9df6-a6ff5e409fe9" containerID="1ceeda90e86c36df6ba2076a315a4cfba4c20fca9a547843260ca82bb556d260" exitCode=0 Jul 23 00:14:22.256270 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:22.256256 2562 generic.go:356] "Generic (PLEG): container finished" podID="bd760290-ebfd-4cdb-9df6-a6ff5e409fe9" containerID="6c1751ed08f9c10d74456a5cee6ee9c413952984c4aafa1f5915a44a5d5559c6" exitCode=0 Jul 23 00:14:22.256270 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:22.256265 2562 generic.go:356] "Generic (PLEG): container finished" podID="bd760290-ebfd-4cdb-9df6-a6ff5e409fe9" containerID="a1a82725bb2b176dfb9ddda01fd36435b0f40c912971e1a3612cf7ddf2c48407" exitCode=0 Jul 23 00:14:22.256270 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:22.256271 2562 generic.go:356] "Generic (PLEG): container finished" podID="bd760290-ebfd-4cdb-9df6-a6ff5e409fe9" containerID="3dd3144834eecb1afad31e508d0d658cfd70859f028f6367252b9f75e17eb6d7" exitCode=0 Jul 23 00:14:22.256576 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:22.256301 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9","Type":"ContainerDied","Data":"1ceeda90e86c36df6ba2076a315a4cfba4c20fca9a547843260ca82bb556d260"} Jul 23 00:14:22.256576 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:22.256404 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9","Type":"ContainerDied","Data":"6c1751ed08f9c10d74456a5cee6ee9c413952984c4aafa1f5915a44a5d5559c6"} Jul 23 00:14:22.256576 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:22.256445 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9","Type":"ContainerDied","Data":"a1a82725bb2b176dfb9ddda01fd36435b0f40c912971e1a3612cf7ddf2c48407"} Jul 23 00:14:22.256576 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:22.256477 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9","Type":"ContainerDied","Data":"3dd3144834eecb1afad31e508d0d658cfd70859f028f6367252b9f75e17eb6d7"} Jul 23 00:14:22.258043 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:22.258017 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kfgzz" event={"ID":"5a9bdee1-a493-4223-a7b6-3ae1f5024eb6","Type":"ContainerStarted","Data":"c00a7c7106b2adfd46ce927cab47cf58e3ba9d722e0a9d9103ab36f0d6a461a9"} Jul 23 00:14:22.258173 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:22.258058 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kfgzz" event={"ID":"5a9bdee1-a493-4223-a7b6-3ae1f5024eb6","Type":"ContainerStarted","Data":"c0f60e609072a35e86390d05b4aad70a2f8d042baa38c345ca52052d8a504a54"} Jul 23 00:14:22.274792 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:22.274716 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-kfgzz" podStartSLOduration=374.309852766 podStartE2EDuration="6m15.274699209s" totalImagesPullingTime="964.846443ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:08:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:14:22.274011791 +0000 UTC m=+376.069518669" watchObservedRunningTime="2026-07-23 00:14:22.274699209 +0000 UTC m=+376.070206067" Jul 23 00:14:23.033637 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.033611 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:14:23.142930 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.142836 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-tls-assets\") pod \"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9\" (UID: \"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9\") " Jul 23 00:14:23.142930 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.142880 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9\" (UID: \"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9\") " Jul 23 00:14:23.142930 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.142906 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-config-volume\") pod \"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9\" (UID: \"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9\") " Jul 23 00:14:23.142930 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.142931 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-secret-alertmanager-kube-rbac-proxy-web\") pod \"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9\" (UID: \"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9\") " Jul 23 00:14:23.143268 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.142967 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-secret-alertmanager-kube-rbac-proxy\") pod \"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9\" (UID: \"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9\") " Jul 23 00:14:23.143268 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.143005 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-alertmanager-main-db\") pod \"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9\" (UID: \"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9\") " Jul 23 00:14:23.143268 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.143034 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-alertmanager-trusted-ca-bundle\") pod \"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9\" (UID: \"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9\") " Jul 23 00:14:23.143268 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.143094 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4rxp4\" (UniqueName: \"kubernetes.io/projected/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-kube-api-access-4rxp4\") pod \"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9\" (UID: \"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9\") " Jul 23 00:14:23.143268 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.143124 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-config-out\") pod \"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9\" (UID: \"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9\") " Jul 23 00:14:23.143268 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.143156 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-cluster-tls-config\") pod \"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9\" (UID: \"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9\") " Jul 23 00:14:23.143268 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.143185 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-secret-alertmanager-main-tls\") pod \"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9\" (UID: \"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9\") " Jul 23 00:14:23.143268 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.143219 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-metrics-client-ca\") pod \"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9\" (UID: \"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9\") " Jul 23 00:14:23.143268 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.143249 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-web-config\") pod \"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9\" (UID: \"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9\") " Jul 23 00:14:23.143706 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.143498 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "bd760290-ebfd-4cdb-9df6-a6ff5e409fe9" (UID: "bd760290-ebfd-4cdb-9df6-a6ff5e409fe9"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:14:23.143706 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.143620 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "bd760290-ebfd-4cdb-9df6-a6ff5e409fe9" (UID: "bd760290-ebfd-4cdb-9df6-a6ff5e409fe9"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:14:23.144367 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.144339 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "bd760290-ebfd-4cdb-9df6-a6ff5e409fe9" (UID: "bd760290-ebfd-4cdb-9df6-a6ff5e409fe9"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:14:23.146003 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.145975 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "bd760290-ebfd-4cdb-9df6-a6ff5e409fe9" (UID: "bd760290-ebfd-4cdb-9df6-a6ff5e409fe9"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:14:23.146282 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.146254 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "bd760290-ebfd-4cdb-9df6-a6ff5e409fe9" (UID: "bd760290-ebfd-4cdb-9df6-a6ff5e409fe9"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:14:23.146453 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.146424 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "bd760290-ebfd-4cdb-9df6-a6ff5e409fe9" (UID: "bd760290-ebfd-4cdb-9df6-a6ff5e409fe9"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:14:23.146956 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.146915 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-config-volume" (OuterVolumeSpecName: "config-volume") pod "bd760290-ebfd-4cdb-9df6-a6ff5e409fe9" (UID: "bd760290-ebfd-4cdb-9df6-a6ff5e409fe9"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:14:23.146956 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.146933 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "bd760290-ebfd-4cdb-9df6-a6ff5e409fe9" (UID: "bd760290-ebfd-4cdb-9df6-a6ff5e409fe9"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:14:23.147108 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.147011 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "bd760290-ebfd-4cdb-9df6-a6ff5e409fe9" (UID: "bd760290-ebfd-4cdb-9df6-a6ff5e409fe9"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:14:23.147400 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.147379 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-config-out" (OuterVolumeSpecName: "config-out") pod "bd760290-ebfd-4cdb-9df6-a6ff5e409fe9" (UID: "bd760290-ebfd-4cdb-9df6-a6ff5e409fe9"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:14:23.147797 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.147747 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-kube-api-access-4rxp4" (OuterVolumeSpecName: "kube-api-access-4rxp4") pod "bd760290-ebfd-4cdb-9df6-a6ff5e409fe9" (UID: "bd760290-ebfd-4cdb-9df6-a6ff5e409fe9"). InnerVolumeSpecName "kube-api-access-4rxp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:14:23.150566 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.150528 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "bd760290-ebfd-4cdb-9df6-a6ff5e409fe9" (UID: "bd760290-ebfd-4cdb-9df6-a6ff5e409fe9"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:14:23.158037 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.158002 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-web-config" (OuterVolumeSpecName: "web-config") pod "bd760290-ebfd-4cdb-9df6-a6ff5e409fe9" (UID: "bd760290-ebfd-4cdb-9df6-a6ff5e409fe9"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:14:23.244593 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.244552 2562 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-tls-assets\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:14:23.244593 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.244588 2562 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:14:23.244593 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.244597 2562 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-config-volume\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:14:23.244593 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.244606 2562 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:14:23.244912 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.244616 2562 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:14:23.244912 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.244626 2562 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-alertmanager-main-db\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:14:23.244912 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.244635 2562 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:14:23.244912 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.244643 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rxp4\" (UniqueName: \"kubernetes.io/projected/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-kube-api-access-4rxp4\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:14:23.244912 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.244651 2562 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-config-out\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:14:23.244912 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.244658 2562 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-cluster-tls-config\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:14:23.244912 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.244667 2562 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-secret-alertmanager-main-tls\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:14:23.244912 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.244676 2562 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-metrics-client-ca\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:14:23.244912 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.244684 2562 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9-web-config\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:14:23.265070 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.265031 2562 generic.go:356] "Generic (PLEG): container finished" podID="bd760290-ebfd-4cdb-9df6-a6ff5e409fe9" containerID="3e67696fdacf467fd1361eef450ae034675ae2ce102d470dbc9ea131ec4b9cb5" exitCode=0 Jul 23 00:14:23.265070 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.265058 2562 generic.go:356] "Generic (PLEG): container finished" podID="bd760290-ebfd-4cdb-9df6-a6ff5e409fe9" containerID="145b0ae1736093328914513480b6c44ff7e7b9c015a2360aaffec97896244fbd" exitCode=0 Jul 23 00:14:23.265303 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.265116 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:14:23.265303 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.265112 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9","Type":"ContainerDied","Data":"3e67696fdacf467fd1361eef450ae034675ae2ce102d470dbc9ea131ec4b9cb5"} Jul 23 00:14:23.265303 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.265202 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9","Type":"ContainerDied","Data":"145b0ae1736093328914513480b6c44ff7e7b9c015a2360aaffec97896244fbd"} Jul 23 00:14:23.265303 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.265245 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bd760290-ebfd-4cdb-9df6-a6ff5e409fe9","Type":"ContainerDied","Data":"854a0bd686a12bc8e62a73daa7d557877a5e66ebf08ae69d47d459e2d878c1b7"} Jul 23 00:14:23.265303 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.265291 2562 scope.go:117] "RemoveContainer" containerID="1ceeda90e86c36df6ba2076a315a4cfba4c20fca9a547843260ca82bb556d260" Jul 23 00:14:23.274571 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.274545 2562 scope.go:117] "RemoveContainer" containerID="3e67696fdacf467fd1361eef450ae034675ae2ce102d470dbc9ea131ec4b9cb5" Jul 23 00:14:23.282428 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.282406 2562 scope.go:117] "RemoveContainer" containerID="6c1751ed08f9c10d74456a5cee6ee9c413952984c4aafa1f5915a44a5d5559c6" Jul 23 00:14:23.289898 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.289872 2562 scope.go:117] "RemoveContainer" containerID="145b0ae1736093328914513480b6c44ff7e7b9c015a2360aaffec97896244fbd" Jul 23 00:14:23.291824 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.291797 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 23 00:14:23.298190 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.298161 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 23 00:14:23.298589 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.298572 2562 scope.go:117] "RemoveContainer" containerID="a1a82725bb2b176dfb9ddda01fd36435b0f40c912971e1a3612cf7ddf2c48407" Jul 23 00:14:23.306625 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.306602 2562 scope.go:117] "RemoveContainer" containerID="3dd3144834eecb1afad31e508d0d658cfd70859f028f6367252b9f75e17eb6d7" Jul 23 00:14:23.315073 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.315052 2562 scope.go:117] "RemoveContainer" containerID="8b6345a2805d2a3717621710c802a01d7e4b1234e4a8655b9497331c04b90104" Jul 23 00:14:23.323213 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.323187 2562 scope.go:117] "RemoveContainer" containerID="1ceeda90e86c36df6ba2076a315a4cfba4c20fca9a547843260ca82bb556d260" Jul 23 00:14:23.323557 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:14:23.323535 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ceeda90e86c36df6ba2076a315a4cfba4c20fca9a547843260ca82bb556d260\": container with ID starting with 1ceeda90e86c36df6ba2076a315a4cfba4c20fca9a547843260ca82bb556d260 not found: ID does not exist" containerID="1ceeda90e86c36df6ba2076a315a4cfba4c20fca9a547843260ca82bb556d260" Jul 23 00:14:23.323634 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.323581 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 23 00:14:23.323692 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.323567 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ceeda90e86c36df6ba2076a315a4cfba4c20fca9a547843260ca82bb556d260"} err="failed to get container status \"1ceeda90e86c36df6ba2076a315a4cfba4c20fca9a547843260ca82bb556d260\": rpc error: code = NotFound desc = could not find container \"1ceeda90e86c36df6ba2076a315a4cfba4c20fca9a547843260ca82bb556d260\": container with ID starting with 1ceeda90e86c36df6ba2076a315a4cfba4c20fca9a547843260ca82bb556d260 not found: ID does not exist" Jul 23 00:14:23.323692 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.323678 2562 scope.go:117] "RemoveContainer" containerID="3e67696fdacf467fd1361eef450ae034675ae2ce102d470dbc9ea131ec4b9cb5" Jul 23 00:14:23.323989 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.323971 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd760290-ebfd-4cdb-9df6-a6ff5e409fe9" containerName="kube-rbac-proxy-web" Jul 23 00:14:23.324072 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.323993 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd760290-ebfd-4cdb-9df6-a6ff5e409fe9" containerName="kube-rbac-proxy-web" Jul 23 00:14:23.324072 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.324005 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd760290-ebfd-4cdb-9df6-a6ff5e409fe9" containerName="kube-rbac-proxy-metric" Jul 23 00:14:23.324072 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.324014 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd760290-ebfd-4cdb-9df6-a6ff5e409fe9" containerName="kube-rbac-proxy-metric" Jul 23 00:14:23.324072 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.324028 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd760290-ebfd-4cdb-9df6-a6ff5e409fe9" containerName="prom-label-proxy" Jul 23 00:14:23.324072 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:14:23.324028 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e67696fdacf467fd1361eef450ae034675ae2ce102d470dbc9ea131ec4b9cb5\": container with ID starting with 3e67696fdacf467fd1361eef450ae034675ae2ce102d470dbc9ea131ec4b9cb5 not found: ID does not exist" containerID="3e67696fdacf467fd1361eef450ae034675ae2ce102d470dbc9ea131ec4b9cb5" Jul 23 00:14:23.324072 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.324038 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd760290-ebfd-4cdb-9df6-a6ff5e409fe9" containerName="prom-label-proxy" Jul 23 00:14:23.324072 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.324052 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd760290-ebfd-4cdb-9df6-a6ff5e409fe9" containerName="alertmanager" Jul 23 00:14:23.324072 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.324061 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd760290-ebfd-4cdb-9df6-a6ff5e409fe9" containerName="alertmanager" Jul 23 00:14:23.324072 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.324071 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd760290-ebfd-4cdb-9df6-a6ff5e409fe9" containerName="kube-rbac-proxy" Jul 23 00:14:23.324414 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.324080 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd760290-ebfd-4cdb-9df6-a6ff5e409fe9" containerName="kube-rbac-proxy" Jul 23 00:14:23.324414 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.324089 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd760290-ebfd-4cdb-9df6-a6ff5e409fe9" containerName="init-config-reloader" Jul 23 00:14:23.324414 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.324098 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd760290-ebfd-4cdb-9df6-a6ff5e409fe9" containerName="init-config-reloader" Jul 23 00:14:23.324414 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.324045 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e67696fdacf467fd1361eef450ae034675ae2ce102d470dbc9ea131ec4b9cb5"} err="failed to get container status \"3e67696fdacf467fd1361eef450ae034675ae2ce102d470dbc9ea131ec4b9cb5\": rpc error: code = NotFound desc = could not find container \"3e67696fdacf467fd1361eef450ae034675ae2ce102d470dbc9ea131ec4b9cb5\": container with ID starting with 3e67696fdacf467fd1361eef450ae034675ae2ce102d470dbc9ea131ec4b9cb5 not found: ID does not exist" Jul 23 00:14:23.324414 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.324122 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd760290-ebfd-4cdb-9df6-a6ff5e409fe9" containerName="config-reloader" Jul 23 00:14:23.324414 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.324130 2562 scope.go:117] "RemoveContainer" containerID="6c1751ed08f9c10d74456a5cee6ee9c413952984c4aafa1f5915a44a5d5559c6" Jul 23 00:14:23.324414 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.324131 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd760290-ebfd-4cdb-9df6-a6ff5e409fe9" containerName="config-reloader" Jul 23 00:14:23.324414 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.324254 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd760290-ebfd-4cdb-9df6-a6ff5e409fe9" containerName="kube-rbac-proxy-web" Jul 23 00:14:23.324414 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.324265 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd760290-ebfd-4cdb-9df6-a6ff5e409fe9" containerName="kube-rbac-proxy" Jul 23 00:14:23.324414 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.324272 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd760290-ebfd-4cdb-9df6-a6ff5e409fe9" containerName="config-reloader" Jul 23 00:14:23.324414 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.324280 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd760290-ebfd-4cdb-9df6-a6ff5e409fe9" containerName="kube-rbac-proxy-metric" Jul 23 00:14:23.324414 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.324288 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd760290-ebfd-4cdb-9df6-a6ff5e409fe9" containerName="alertmanager" Jul 23 00:14:23.324414 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.324297 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd760290-ebfd-4cdb-9df6-a6ff5e409fe9" containerName="prom-label-proxy" Jul 23 00:14:23.324414 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:14:23.324398 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c1751ed08f9c10d74456a5cee6ee9c413952984c4aafa1f5915a44a5d5559c6\": container with ID starting with 6c1751ed08f9c10d74456a5cee6ee9c413952984c4aafa1f5915a44a5d5559c6 not found: ID does not exist" containerID="6c1751ed08f9c10d74456a5cee6ee9c413952984c4aafa1f5915a44a5d5559c6" Jul 23 00:14:23.325005 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.324425 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c1751ed08f9c10d74456a5cee6ee9c413952984c4aafa1f5915a44a5d5559c6"} err="failed to get container status \"6c1751ed08f9c10d74456a5cee6ee9c413952984c4aafa1f5915a44a5d5559c6\": rpc error: code = NotFound desc = could not find container \"6c1751ed08f9c10d74456a5cee6ee9c413952984c4aafa1f5915a44a5d5559c6\": container with ID starting with 6c1751ed08f9c10d74456a5cee6ee9c413952984c4aafa1f5915a44a5d5559c6 not found: ID does not exist" Jul 23 00:14:23.325005 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.324448 2562 scope.go:117] "RemoveContainer" containerID="145b0ae1736093328914513480b6c44ff7e7b9c015a2360aaffec97896244fbd" Jul 23 00:14:23.325005 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:14:23.324688 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"145b0ae1736093328914513480b6c44ff7e7b9c015a2360aaffec97896244fbd\": container with ID starting with 145b0ae1736093328914513480b6c44ff7e7b9c015a2360aaffec97896244fbd not found: ID does not exist" containerID="145b0ae1736093328914513480b6c44ff7e7b9c015a2360aaffec97896244fbd" Jul 23 00:14:23.325005 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.324711 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"145b0ae1736093328914513480b6c44ff7e7b9c015a2360aaffec97896244fbd"} err="failed to get container status \"145b0ae1736093328914513480b6c44ff7e7b9c015a2360aaffec97896244fbd\": rpc error: code = NotFound desc = could not find container \"145b0ae1736093328914513480b6c44ff7e7b9c015a2360aaffec97896244fbd\": container with ID starting with 145b0ae1736093328914513480b6c44ff7e7b9c015a2360aaffec97896244fbd not found: ID does not exist" Jul 23 00:14:23.325005 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.324732 2562 scope.go:117] "RemoveContainer" containerID="a1a82725bb2b176dfb9ddda01fd36435b0f40c912971e1a3612cf7ddf2c48407" Jul 23 00:14:23.325005 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:14:23.324988 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1a82725bb2b176dfb9ddda01fd36435b0f40c912971e1a3612cf7ddf2c48407\": container with ID starting with a1a82725bb2b176dfb9ddda01fd36435b0f40c912971e1a3612cf7ddf2c48407 not found: ID does not exist" containerID="a1a82725bb2b176dfb9ddda01fd36435b0f40c912971e1a3612cf7ddf2c48407" Jul 23 00:14:23.325195 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.325008 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1a82725bb2b176dfb9ddda01fd36435b0f40c912971e1a3612cf7ddf2c48407"} err="failed to get container status \"a1a82725bb2b176dfb9ddda01fd36435b0f40c912971e1a3612cf7ddf2c48407\": rpc error: code = NotFound desc = could not find container \"a1a82725bb2b176dfb9ddda01fd36435b0f40c912971e1a3612cf7ddf2c48407\": container with ID starting with a1a82725bb2b176dfb9ddda01fd36435b0f40c912971e1a3612cf7ddf2c48407 not found: ID does not exist" Jul 23 00:14:23.325195 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.325023 2562 scope.go:117] "RemoveContainer" containerID="3dd3144834eecb1afad31e508d0d658cfd70859f028f6367252b9f75e17eb6d7" Jul 23 00:14:23.325314 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:14:23.325296 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dd3144834eecb1afad31e508d0d658cfd70859f028f6367252b9f75e17eb6d7\": container with ID starting with 3dd3144834eecb1afad31e508d0d658cfd70859f028f6367252b9f75e17eb6d7 not found: ID does not exist" containerID="3dd3144834eecb1afad31e508d0d658cfd70859f028f6367252b9f75e17eb6d7" Jul 23 00:14:23.325350 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.325320 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dd3144834eecb1afad31e508d0d658cfd70859f028f6367252b9f75e17eb6d7"} err="failed to get container status \"3dd3144834eecb1afad31e508d0d658cfd70859f028f6367252b9f75e17eb6d7\": rpc error: code = NotFound desc = could not find container \"3dd3144834eecb1afad31e508d0d658cfd70859f028f6367252b9f75e17eb6d7\": container with ID starting with 3dd3144834eecb1afad31e508d0d658cfd70859f028f6367252b9f75e17eb6d7 not found: ID does not exist" Jul 23 00:14:23.325350 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.325336 2562 scope.go:117] "RemoveContainer" containerID="8b6345a2805d2a3717621710c802a01d7e4b1234e4a8655b9497331c04b90104" Jul 23 00:14:23.325576 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:14:23.325554 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b6345a2805d2a3717621710c802a01d7e4b1234e4a8655b9497331c04b90104\": container with ID starting with 8b6345a2805d2a3717621710c802a01d7e4b1234e4a8655b9497331c04b90104 not found: ID does not exist" containerID="8b6345a2805d2a3717621710c802a01d7e4b1234e4a8655b9497331c04b90104" Jul 23 00:14:23.325660 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.325582 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b6345a2805d2a3717621710c802a01d7e4b1234e4a8655b9497331c04b90104"} err="failed to get container status \"8b6345a2805d2a3717621710c802a01d7e4b1234e4a8655b9497331c04b90104\": rpc error: code = NotFound desc = could not find container \"8b6345a2805d2a3717621710c802a01d7e4b1234e4a8655b9497331c04b90104\": container with ID starting with 8b6345a2805d2a3717621710c802a01d7e4b1234e4a8655b9497331c04b90104 not found: ID does not exist" Jul 23 00:14:23.325660 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.325602 2562 scope.go:117] "RemoveContainer" containerID="1ceeda90e86c36df6ba2076a315a4cfba4c20fca9a547843260ca82bb556d260" Jul 23 00:14:23.325858 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.325837 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ceeda90e86c36df6ba2076a315a4cfba4c20fca9a547843260ca82bb556d260"} err="failed to get container status \"1ceeda90e86c36df6ba2076a315a4cfba4c20fca9a547843260ca82bb556d260\": rpc error: code = NotFound desc = could not find container \"1ceeda90e86c36df6ba2076a315a4cfba4c20fca9a547843260ca82bb556d260\": container with ID starting with 1ceeda90e86c36df6ba2076a315a4cfba4c20fca9a547843260ca82bb556d260 not found: ID does not exist" Jul 23 00:14:23.325910 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.325860 2562 scope.go:117] "RemoveContainer" containerID="3e67696fdacf467fd1361eef450ae034675ae2ce102d470dbc9ea131ec4b9cb5" Jul 23 00:14:23.326066 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.326049 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e67696fdacf467fd1361eef450ae034675ae2ce102d470dbc9ea131ec4b9cb5"} err="failed to get container status \"3e67696fdacf467fd1361eef450ae034675ae2ce102d470dbc9ea131ec4b9cb5\": rpc error: code = NotFound desc = could not find container \"3e67696fdacf467fd1361eef450ae034675ae2ce102d470dbc9ea131ec4b9cb5\": container with ID starting with 3e67696fdacf467fd1361eef450ae034675ae2ce102d470dbc9ea131ec4b9cb5 not found: ID does not exist" Jul 23 00:14:23.326139 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.326067 2562 scope.go:117] "RemoveContainer" containerID="6c1751ed08f9c10d74456a5cee6ee9c413952984c4aafa1f5915a44a5d5559c6" Jul 23 00:14:23.326285 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.326269 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c1751ed08f9c10d74456a5cee6ee9c413952984c4aafa1f5915a44a5d5559c6"} err="failed to get container status \"6c1751ed08f9c10d74456a5cee6ee9c413952984c4aafa1f5915a44a5d5559c6\": rpc error: code = NotFound desc = could not find container \"6c1751ed08f9c10d74456a5cee6ee9c413952984c4aafa1f5915a44a5d5559c6\": container with ID starting with 6c1751ed08f9c10d74456a5cee6ee9c413952984c4aafa1f5915a44a5d5559c6 not found: ID does not exist" Jul 23 00:14:23.326349 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.326286 2562 scope.go:117] "RemoveContainer" containerID="145b0ae1736093328914513480b6c44ff7e7b9c015a2360aaffec97896244fbd" Jul 23 00:14:23.326524 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.326503 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"145b0ae1736093328914513480b6c44ff7e7b9c015a2360aaffec97896244fbd"} err="failed to get container status \"145b0ae1736093328914513480b6c44ff7e7b9c015a2360aaffec97896244fbd\": rpc error: code = NotFound desc = could not find container \"145b0ae1736093328914513480b6c44ff7e7b9c015a2360aaffec97896244fbd\": container with ID starting with 145b0ae1736093328914513480b6c44ff7e7b9c015a2360aaffec97896244fbd not found: ID does not exist" Jul 23 00:14:23.326571 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.326527 2562 scope.go:117] "RemoveContainer" containerID="a1a82725bb2b176dfb9ddda01fd36435b0f40c912971e1a3612cf7ddf2c48407" Jul 23 00:14:23.326742 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.326724 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1a82725bb2b176dfb9ddda01fd36435b0f40c912971e1a3612cf7ddf2c48407"} err="failed to get container status \"a1a82725bb2b176dfb9ddda01fd36435b0f40c912971e1a3612cf7ddf2c48407\": rpc error: code = NotFound desc = could not find container \"a1a82725bb2b176dfb9ddda01fd36435b0f40c912971e1a3612cf7ddf2c48407\": container with ID starting with a1a82725bb2b176dfb9ddda01fd36435b0f40c912971e1a3612cf7ddf2c48407 not found: ID does not exist" Jul 23 00:14:23.326827 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.326743 2562 scope.go:117] "RemoveContainer" containerID="3dd3144834eecb1afad31e508d0d658cfd70859f028f6367252b9f75e17eb6d7" Jul 23 00:14:23.327023 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.327001 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dd3144834eecb1afad31e508d0d658cfd70859f028f6367252b9f75e17eb6d7"} err="failed to get container status \"3dd3144834eecb1afad31e508d0d658cfd70859f028f6367252b9f75e17eb6d7\": rpc error: code = NotFound desc = could not find container \"3dd3144834eecb1afad31e508d0d658cfd70859f028f6367252b9f75e17eb6d7\": container with ID starting with 3dd3144834eecb1afad31e508d0d658cfd70859f028f6367252b9f75e17eb6d7 not found: ID does not exist" Jul 23 00:14:23.327023 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.327021 2562 scope.go:117] "RemoveContainer" containerID="8b6345a2805d2a3717621710c802a01d7e4b1234e4a8655b9497331c04b90104" Jul 23 00:14:23.327214 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.327196 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b6345a2805d2a3717621710c802a01d7e4b1234e4a8655b9497331c04b90104"} err="failed to get container status \"8b6345a2805d2a3717621710c802a01d7e4b1234e4a8655b9497331c04b90104\": rpc error: code = NotFound desc = could not find container \"8b6345a2805d2a3717621710c802a01d7e4b1234e4a8655b9497331c04b90104\": container with ID starting with 8b6345a2805d2a3717621710c802a01d7e4b1234e4a8655b9497331c04b90104 not found: ID does not exist" Jul 23 00:14:23.329488 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.329469 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:14:23.332430 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.332381 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 23 00:14:23.332430 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.332381 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 23 00:14:23.332635 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.332473 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 23 00:14:23.332635 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.332588 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 23 00:14:23.332743 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.332693 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 23 00:14:23.332818 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.332807 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 23 00:14:23.332916 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.332893 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-jvpp4\"" Jul 23 00:14:23.332979 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.332925 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 23 00:14:23.333030 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.332975 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 23 00:14:23.337903 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.337881 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 23 00:14:23.342080 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.342053 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 23 00:14:23.345673 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.345647 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/60761a85-20d1-4175-a35b-34bd82651f37-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"60761a85-20d1-4175-a35b-34bd82651f37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:14:23.345796 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.345680 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60761a85-20d1-4175-a35b-34bd82651f37-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"60761a85-20d1-4175-a35b-34bd82651f37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:14:23.345796 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.345705 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/60761a85-20d1-4175-a35b-34bd82651f37-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"60761a85-20d1-4175-a35b-34bd82651f37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:14:23.345796 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.345722 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/60761a85-20d1-4175-a35b-34bd82651f37-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"60761a85-20d1-4175-a35b-34bd82651f37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:14:23.345908 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.345808 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60761a85-20d1-4175-a35b-34bd82651f37-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"60761a85-20d1-4175-a35b-34bd82651f37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:14:23.345908 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.345851 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/60761a85-20d1-4175-a35b-34bd82651f37-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"60761a85-20d1-4175-a35b-34bd82651f37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:14:23.345908 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.345877 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/60761a85-20d1-4175-a35b-34bd82651f37-tls-assets\") pod \"alertmanager-main-0\" (UID: \"60761a85-20d1-4175-a35b-34bd82651f37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:14:23.345908 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.345895 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/60761a85-20d1-4175-a35b-34bd82651f37-config-out\") pod \"alertmanager-main-0\" (UID: \"60761a85-20d1-4175-a35b-34bd82651f37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:14:23.346056 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.345927 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/60761a85-20d1-4175-a35b-34bd82651f37-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"60761a85-20d1-4175-a35b-34bd82651f37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:14:23.346056 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.345958 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/60761a85-20d1-4175-a35b-34bd82651f37-web-config\") pod \"alertmanager-main-0\" (UID: \"60761a85-20d1-4175-a35b-34bd82651f37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:14:23.346056 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.345984 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jn9g\" (UniqueName: \"kubernetes.io/projected/60761a85-20d1-4175-a35b-34bd82651f37-kube-api-access-9jn9g\") pod \"alertmanager-main-0\" (UID: \"60761a85-20d1-4175-a35b-34bd82651f37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:14:23.346056 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.346024 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/60761a85-20d1-4175-a35b-34bd82651f37-config-volume\") pod \"alertmanager-main-0\" (UID: \"60761a85-20d1-4175-a35b-34bd82651f37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:14:23.346181 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.346076 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/60761a85-20d1-4175-a35b-34bd82651f37-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"60761a85-20d1-4175-a35b-34bd82651f37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:14:23.446609 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.446508 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/60761a85-20d1-4175-a35b-34bd82651f37-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"60761a85-20d1-4175-a35b-34bd82651f37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:14:23.446609 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.446552 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60761a85-20d1-4175-a35b-34bd82651f37-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"60761a85-20d1-4175-a35b-34bd82651f37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:14:23.446897 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.446646 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/60761a85-20d1-4175-a35b-34bd82651f37-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"60761a85-20d1-4175-a35b-34bd82651f37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:14:23.446897 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.446668 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/60761a85-20d1-4175-a35b-34bd82651f37-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"60761a85-20d1-4175-a35b-34bd82651f37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:14:23.446897 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.446686 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60761a85-20d1-4175-a35b-34bd82651f37-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"60761a85-20d1-4175-a35b-34bd82651f37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:14:23.446897 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.446708 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/60761a85-20d1-4175-a35b-34bd82651f37-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"60761a85-20d1-4175-a35b-34bd82651f37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:14:23.446897 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.446726 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/60761a85-20d1-4175-a35b-34bd82651f37-tls-assets\") pod \"alertmanager-main-0\" (UID: \"60761a85-20d1-4175-a35b-34bd82651f37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:14:23.446897 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.446843 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/60761a85-20d1-4175-a35b-34bd82651f37-config-out\") pod \"alertmanager-main-0\" (UID: \"60761a85-20d1-4175-a35b-34bd82651f37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:14:23.447194 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.446898 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/60761a85-20d1-4175-a35b-34bd82651f37-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"60761a85-20d1-4175-a35b-34bd82651f37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:14:23.447194 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.446928 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/60761a85-20d1-4175-a35b-34bd82651f37-web-config\") pod \"alertmanager-main-0\" (UID: \"60761a85-20d1-4175-a35b-34bd82651f37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:14:23.447194 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.446954 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jn9g\" (UniqueName: \"kubernetes.io/projected/60761a85-20d1-4175-a35b-34bd82651f37-kube-api-access-9jn9g\") pod \"alertmanager-main-0\" (UID: \"60761a85-20d1-4175-a35b-34bd82651f37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:14:23.447194 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.446987 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/60761a85-20d1-4175-a35b-34bd82651f37-config-volume\") pod \"alertmanager-main-0\" (UID: \"60761a85-20d1-4175-a35b-34bd82651f37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:14:23.447194 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.447031 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/60761a85-20d1-4175-a35b-34bd82651f37-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"60761a85-20d1-4175-a35b-34bd82651f37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:14:23.447194 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.447140 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/60761a85-20d1-4175-a35b-34bd82651f37-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"60761a85-20d1-4175-a35b-34bd82651f37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:14:23.447736 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.447683 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60761a85-20d1-4175-a35b-34bd82651f37-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"60761a85-20d1-4175-a35b-34bd82651f37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:14:23.448510 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.448479 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60761a85-20d1-4175-a35b-34bd82651f37-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"60761a85-20d1-4175-a35b-34bd82651f37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:14:23.450029 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.450001 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/60761a85-20d1-4175-a35b-34bd82651f37-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"60761a85-20d1-4175-a35b-34bd82651f37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:14:23.450121 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.450001 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/60761a85-20d1-4175-a35b-34bd82651f37-config-out\") pod \"alertmanager-main-0\" (UID: \"60761a85-20d1-4175-a35b-34bd82651f37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:14:23.450521 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.450450 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/60761a85-20d1-4175-a35b-34bd82651f37-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"60761a85-20d1-4175-a35b-34bd82651f37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:14:23.450521 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.450476 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/60761a85-20d1-4175-a35b-34bd82651f37-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"60761a85-20d1-4175-a35b-34bd82651f37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:14:23.450521 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.450484 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/60761a85-20d1-4175-a35b-34bd82651f37-tls-assets\") pod \"alertmanager-main-0\" (UID: \"60761a85-20d1-4175-a35b-34bd82651f37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:14:23.450739 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.450608 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/60761a85-20d1-4175-a35b-34bd82651f37-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"60761a85-20d1-4175-a35b-34bd82651f37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:14:23.450837 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.450822 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/60761a85-20d1-4175-a35b-34bd82651f37-config-volume\") pod \"alertmanager-main-0\" (UID: \"60761a85-20d1-4175-a35b-34bd82651f37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:14:23.450889 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.450868 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/60761a85-20d1-4175-a35b-34bd82651f37-web-config\") pod \"alertmanager-main-0\" (UID: \"60761a85-20d1-4175-a35b-34bd82651f37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:14:23.451618 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.451596 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/60761a85-20d1-4175-a35b-34bd82651f37-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"60761a85-20d1-4175-a35b-34bd82651f37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:14:23.455315 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.455280 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jn9g\" (UniqueName: \"kubernetes.io/projected/60761a85-20d1-4175-a35b-34bd82651f37-kube-api-access-9jn9g\") pod \"alertmanager-main-0\" (UID: \"60761a85-20d1-4175-a35b-34bd82651f37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:14:23.639521 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.639480 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:14:23.789571 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:23.789534 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 23 00:14:23.792670 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:14:23.792641 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60761a85_20d1_4175_a35b_34bd82651f37.slice/crio-787aa26d6a0f66440ea1a50f8418e81aa24c34a2a3d8820d1bc9dd58b88264de WatchSource:0}: Error finding container 787aa26d6a0f66440ea1a50f8418e81aa24c34a2a3d8820d1bc9dd58b88264de: Status 404 returned error can't find the container with id 787aa26d6a0f66440ea1a50f8418e81aa24c34a2a3d8820d1bc9dd58b88264de Jul 23 00:14:24.272585 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:24.272544 2562 generic.go:356] "Generic (PLEG): container finished" podID="60761a85-20d1-4175-a35b-34bd82651f37" containerID="27af28b4136f88c7d2e670b24d170532dc09ec8e0596000a3539b12dcce0a955" exitCode=0 Jul 23 00:14:24.273041 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:24.272633 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"60761a85-20d1-4175-a35b-34bd82651f37","Type":"ContainerDied","Data":"27af28b4136f88c7d2e670b24d170532dc09ec8e0596000a3539b12dcce0a955"} Jul 23 00:14:24.273041 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:24.272704 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"60761a85-20d1-4175-a35b-34bd82651f37","Type":"ContainerStarted","Data":"787aa26d6a0f66440ea1a50f8418e81aa24c34a2a3d8820d1bc9dd58b88264de"} Jul 23 00:14:25.006273 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:25.006182 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd760290-ebfd-4cdb-9df6-a6ff5e409fe9" path="/var/lib/kubelet/pods/bd760290-ebfd-4cdb-9df6-a6ff5e409fe9/volumes" Jul 23 00:14:25.279110 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:25.279077 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"60761a85-20d1-4175-a35b-34bd82651f37","Type":"ContainerStarted","Data":"527826e42c2750d308ad589b43ac3def0791e57f82be605fae14f57655b1e33c"} Jul 23 00:14:25.279512 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:25.279150 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"60761a85-20d1-4175-a35b-34bd82651f37","Type":"ContainerStarted","Data":"de180a21b0acf9a008ee1f8b68a894b1435f967d70576aa689017c09c7928af9"} Jul 23 00:14:25.279512 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:25.279179 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"60761a85-20d1-4175-a35b-34bd82651f37","Type":"ContainerStarted","Data":"51e6748d6e4d7f89ee45ddee67000332964f08a30d47f04e74d8c1d080eb1f30"} Jul 23 00:14:25.279512 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:25.279206 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"60761a85-20d1-4175-a35b-34bd82651f37","Type":"ContainerStarted","Data":"2292861c62a70729a5a2db468c16b41ef416b5e2c43d3d17a93bced58f47a832"} Jul 23 00:14:25.279512 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:25.279234 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"60761a85-20d1-4175-a35b-34bd82651f37","Type":"ContainerStarted","Data":"c4da75b7e943669382e6407bb31a1f28532799cd3b4e4f459769d3606292763d"} Jul 23 00:14:25.279512 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:25.279260 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"60761a85-20d1-4175-a35b-34bd82651f37","Type":"ContainerStarted","Data":"ae54a04459367b037258bbfffdf78a262ff565a6e4a2109ff142796d732a5c51"} Jul 23 00:14:25.306627 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:25.306572 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.289825381 podStartE2EDuration="2.306550293s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.724912ms" isStatefulPod=false podCreationTimestamp="2026-07-23 00:14:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:14:25.304489942 +0000 UTC m=+379.099996801" watchObservedRunningTime="2026-07-23 00:14:25.306550293 +0000 UTC m=+379.102057219" Jul 23 00:14:25.810111 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:25.810076 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-96859b58f-2d4mn"] Jul 23 00:14:25.813832 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:25.813811 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-96859b58f-2d4mn" Jul 23 00:14:25.816453 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:25.816416 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 23 00:14:25.816453 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:25.816447 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 23 00:14:25.816803 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:25.816781 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-n7tvk\"" Jul 23 00:14:25.816909 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:25.816819 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 23 00:14:25.816909 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:25.816830 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 23 00:14:25.816909 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:25.816825 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 23 00:14:25.822221 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:25.822195 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 23 00:14:25.827389 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:25.827360 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-96859b58f-2d4mn"] Jul 23 00:14:25.866183 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:25.866134 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/fda29169-d2fd-464b-87a6-ff7ce5dba70d-telemeter-client-tls\") pod \"telemeter-client-96859b58f-2d4mn\" (UID: \"fda29169-d2fd-464b-87a6-ff7ce5dba70d\") " pod="openshift-monitoring/telemeter-client-96859b58f-2d4mn" Jul 23 00:14:25.866377 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:25.866207 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/fda29169-d2fd-464b-87a6-ff7ce5dba70d-federate-client-tls\") pod \"telemeter-client-96859b58f-2d4mn\" (UID: \"fda29169-d2fd-464b-87a6-ff7ce5dba70d\") " pod="openshift-monitoring/telemeter-client-96859b58f-2d4mn" Jul 23 00:14:25.866377 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:25.866236 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/fda29169-d2fd-464b-87a6-ff7ce5dba70d-secret-telemeter-client\") pod \"telemeter-client-96859b58f-2d4mn\" (UID: \"fda29169-d2fd-464b-87a6-ff7ce5dba70d\") " pod="openshift-monitoring/telemeter-client-96859b58f-2d4mn" Jul 23 00:14:25.866377 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:25.866253 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klszx\" (UniqueName: \"kubernetes.io/projected/fda29169-d2fd-464b-87a6-ff7ce5dba70d-kube-api-access-klszx\") pod \"telemeter-client-96859b58f-2d4mn\" (UID: \"fda29169-d2fd-464b-87a6-ff7ce5dba70d\") " pod="openshift-monitoring/telemeter-client-96859b58f-2d4mn" Jul 23 00:14:25.866377 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:25.866293 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fda29169-d2fd-464b-87a6-ff7ce5dba70d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-96859b58f-2d4mn\" (UID: \"fda29169-d2fd-464b-87a6-ff7ce5dba70d\") " pod="openshift-monitoring/telemeter-client-96859b58f-2d4mn" Jul 23 00:14:25.866576 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:25.866383 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fda29169-d2fd-464b-87a6-ff7ce5dba70d-serving-certs-ca-bundle\") pod \"telemeter-client-96859b58f-2d4mn\" (UID: \"fda29169-d2fd-464b-87a6-ff7ce5dba70d\") " pod="openshift-monitoring/telemeter-client-96859b58f-2d4mn" Jul 23 00:14:25.866576 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:25.866440 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fda29169-d2fd-464b-87a6-ff7ce5dba70d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-96859b58f-2d4mn\" (UID: \"fda29169-d2fd-464b-87a6-ff7ce5dba70d\") " pod="openshift-monitoring/telemeter-client-96859b58f-2d4mn" Jul 23 00:14:25.866576 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:25.866472 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fda29169-d2fd-464b-87a6-ff7ce5dba70d-metrics-client-ca\") pod \"telemeter-client-96859b58f-2d4mn\" (UID: \"fda29169-d2fd-464b-87a6-ff7ce5dba70d\") " pod="openshift-monitoring/telemeter-client-96859b58f-2d4mn" Jul 23 00:14:25.967621 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:25.967585 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fda29169-d2fd-464b-87a6-ff7ce5dba70d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-96859b58f-2d4mn\" (UID: \"fda29169-d2fd-464b-87a6-ff7ce5dba70d\") " pod="openshift-monitoring/telemeter-client-96859b58f-2d4mn" Jul 23 00:14:25.967854 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:25.967631 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fda29169-d2fd-464b-87a6-ff7ce5dba70d-serving-certs-ca-bundle\") pod \"telemeter-client-96859b58f-2d4mn\" (UID: \"fda29169-d2fd-464b-87a6-ff7ce5dba70d\") " pod="openshift-monitoring/telemeter-client-96859b58f-2d4mn" Jul 23 00:14:25.967854 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:25.967657 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fda29169-d2fd-464b-87a6-ff7ce5dba70d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-96859b58f-2d4mn\" (UID: \"fda29169-d2fd-464b-87a6-ff7ce5dba70d\") " pod="openshift-monitoring/telemeter-client-96859b58f-2d4mn" Jul 23 00:14:25.967854 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:25.967680 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fda29169-d2fd-464b-87a6-ff7ce5dba70d-metrics-client-ca\") pod \"telemeter-client-96859b58f-2d4mn\" (UID: \"fda29169-d2fd-464b-87a6-ff7ce5dba70d\") " pod="openshift-monitoring/telemeter-client-96859b58f-2d4mn" Jul 23 00:14:25.967854 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:25.967734 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/fda29169-d2fd-464b-87a6-ff7ce5dba70d-telemeter-client-tls\") pod \"telemeter-client-96859b58f-2d4mn\" (UID: \"fda29169-d2fd-464b-87a6-ff7ce5dba70d\") " pod="openshift-monitoring/telemeter-client-96859b58f-2d4mn" Jul 23 00:14:25.967854 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:25.967851 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/fda29169-d2fd-464b-87a6-ff7ce5dba70d-federate-client-tls\") pod \"telemeter-client-96859b58f-2d4mn\" (UID: \"fda29169-d2fd-464b-87a6-ff7ce5dba70d\") " pod="openshift-monitoring/telemeter-client-96859b58f-2d4mn" Jul 23 00:14:25.968120 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:25.967885 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/fda29169-d2fd-464b-87a6-ff7ce5dba70d-secret-telemeter-client\") pod \"telemeter-client-96859b58f-2d4mn\" (UID: \"fda29169-d2fd-464b-87a6-ff7ce5dba70d\") " pod="openshift-monitoring/telemeter-client-96859b58f-2d4mn" Jul 23 00:14:25.968120 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:25.967907 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klszx\" (UniqueName: \"kubernetes.io/projected/fda29169-d2fd-464b-87a6-ff7ce5dba70d-kube-api-access-klszx\") pod \"telemeter-client-96859b58f-2d4mn\" (UID: \"fda29169-d2fd-464b-87a6-ff7ce5dba70d\") " pod="openshift-monitoring/telemeter-client-96859b58f-2d4mn" Jul 23 00:14:25.968664 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:25.968610 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fda29169-d2fd-464b-87a6-ff7ce5dba70d-metrics-client-ca\") pod \"telemeter-client-96859b58f-2d4mn\" (UID: \"fda29169-d2fd-464b-87a6-ff7ce5dba70d\") " pod="openshift-monitoring/telemeter-client-96859b58f-2d4mn" Jul 23 00:14:25.968800 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:25.968700 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fda29169-d2fd-464b-87a6-ff7ce5dba70d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-96859b58f-2d4mn\" (UID: \"fda29169-d2fd-464b-87a6-ff7ce5dba70d\") " pod="openshift-monitoring/telemeter-client-96859b58f-2d4mn" Jul 23 00:14:25.968896 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:25.968871 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fda29169-d2fd-464b-87a6-ff7ce5dba70d-serving-certs-ca-bundle\") pod \"telemeter-client-96859b58f-2d4mn\" (UID: \"fda29169-d2fd-464b-87a6-ff7ce5dba70d\") " pod="openshift-monitoring/telemeter-client-96859b58f-2d4mn" Jul 23 00:14:25.970435 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:25.970404 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/fda29169-d2fd-464b-87a6-ff7ce5dba70d-secret-telemeter-client\") pod \"telemeter-client-96859b58f-2d4mn\" (UID: \"fda29169-d2fd-464b-87a6-ff7ce5dba70d\") " pod="openshift-monitoring/telemeter-client-96859b58f-2d4mn" Jul 23 00:14:25.970530 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:25.970444 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/fda29169-d2fd-464b-87a6-ff7ce5dba70d-federate-client-tls\") pod \"telemeter-client-96859b58f-2d4mn\" (UID: \"fda29169-d2fd-464b-87a6-ff7ce5dba70d\") " pod="openshift-monitoring/telemeter-client-96859b58f-2d4mn" Jul 23 00:14:25.970621 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:25.970606 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/fda29169-d2fd-464b-87a6-ff7ce5dba70d-telemeter-client-tls\") pod \"telemeter-client-96859b58f-2d4mn\" (UID: \"fda29169-d2fd-464b-87a6-ff7ce5dba70d\") " pod="openshift-monitoring/telemeter-client-96859b58f-2d4mn" Jul 23 00:14:25.970736 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:25.970715 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fda29169-d2fd-464b-87a6-ff7ce5dba70d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-96859b58f-2d4mn\" (UID: \"fda29169-d2fd-464b-87a6-ff7ce5dba70d\") " pod="openshift-monitoring/telemeter-client-96859b58f-2d4mn" Jul 23 00:14:25.976884 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:25.976851 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klszx\" (UniqueName: \"kubernetes.io/projected/fda29169-d2fd-464b-87a6-ff7ce5dba70d-kube-api-access-klszx\") pod \"telemeter-client-96859b58f-2d4mn\" (UID: \"fda29169-d2fd-464b-87a6-ff7ce5dba70d\") " pod="openshift-monitoring/telemeter-client-96859b58f-2d4mn" Jul 23 00:14:26.125042 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:26.124947 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-96859b58f-2d4mn" Jul 23 00:14:26.264209 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:26.264179 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-96859b58f-2d4mn"] Jul 23 00:14:26.266288 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:14:26.266260 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfda29169_d2fd_464b_87a6_ff7ce5dba70d.slice/crio-adc9562042fb5b44dabfb7a6973fc3550b8ed4af6fd3c308555d112f5c1d3a20 WatchSource:0}: Error finding container adc9562042fb5b44dabfb7a6973fc3550b8ed4af6fd3c308555d112f5c1d3a20: Status 404 returned error can't find the container with id adc9562042fb5b44dabfb7a6973fc3550b8ed4af6fd3c308555d112f5c1d3a20 Jul 23 00:14:26.285161 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:26.285127 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-96859b58f-2d4mn" event={"ID":"fda29169-d2fd-464b-87a6-ff7ce5dba70d","Type":"ContainerStarted","Data":"adc9562042fb5b44dabfb7a6973fc3550b8ed4af6fd3c308555d112f5c1d3a20"} Jul 23 00:14:29.298971 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:29.298933 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-96859b58f-2d4mn" event={"ID":"fda29169-d2fd-464b-87a6-ff7ce5dba70d","Type":"ContainerStarted","Data":"4898d68813a062b28f6d385ffdf60df3ab51220b46d8ebfd2c03c77ff17773d4"} Jul 23 00:14:29.299485 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:29.299007 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-96859b58f-2d4mn" event={"ID":"fda29169-d2fd-464b-87a6-ff7ce5dba70d","Type":"ContainerStarted","Data":"c581f91f7477f2bb3f926085d7fbfe9ad4b0deb68b46cc6a62ad5312c903c566"} Jul 23 00:14:29.299485 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:29.299034 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-96859b58f-2d4mn" event={"ID":"fda29169-d2fd-464b-87a6-ff7ce5dba70d","Type":"ContainerStarted","Data":"bd5f24d6c7087b8e1316ba1817757d1a9a531e7eaec24af736936769b73d6fcf"} Jul 23 00:14:29.322472 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:14:29.322417 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-96859b58f-2d4mn" podStartSLOduration=1.913842495 podStartE2EDuration="4.322398835s" totalImagesPullingTime="2.40855634s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:14:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:14:29.319472672 +0000 UTC m=+383.114979538" watchObservedRunningTime="2026-07-23 00:14:29.322398835 +0000 UTC m=+383.117905692" Jul 23 00:16:22.904012 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:22.903971 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402mnvr"] Jul 23 00:16:22.907479 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:22.907456 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402mnvr" Jul 23 00:16:22.910438 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:22.910401 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 23 00:16:22.910438 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:22.910402 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-l2sn7\"" Jul 23 00:16:22.911826 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:22.911801 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 23 00:16:22.916445 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:22.916422 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402mnvr"] Jul 23 00:16:22.960290 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:22.960253 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ef55e071-0d79-43e4-b4e1-493b6bbd736b-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402mnvr\" (UID: \"ef55e071-0d79-43e4-b4e1-493b6bbd736b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402mnvr" Jul 23 00:16:22.960474 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:22.960322 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d74r6\" (UniqueName: \"kubernetes.io/projected/ef55e071-0d79-43e4-b4e1-493b6bbd736b-kube-api-access-d74r6\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402mnvr\" (UID: \"ef55e071-0d79-43e4-b4e1-493b6bbd736b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402mnvr" Jul 23 00:16:22.960474 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:22.960368 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef55e071-0d79-43e4-b4e1-493b6bbd736b-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402mnvr\" (UID: \"ef55e071-0d79-43e4-b4e1-493b6bbd736b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402mnvr" Jul 23 00:16:23.061401 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:23.061355 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ef55e071-0d79-43e4-b4e1-493b6bbd736b-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402mnvr\" (UID: \"ef55e071-0d79-43e4-b4e1-493b6bbd736b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402mnvr" Jul 23 00:16:23.061596 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:23.061424 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d74r6\" (UniqueName: \"kubernetes.io/projected/ef55e071-0d79-43e4-b4e1-493b6bbd736b-kube-api-access-d74r6\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402mnvr\" (UID: \"ef55e071-0d79-43e4-b4e1-493b6bbd736b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402mnvr" Jul 23 00:16:23.061596 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:23.061444 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef55e071-0d79-43e4-b4e1-493b6bbd736b-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402mnvr\" (UID: \"ef55e071-0d79-43e4-b4e1-493b6bbd736b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402mnvr" Jul 23 00:16:23.061851 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:23.061835 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef55e071-0d79-43e4-b4e1-493b6bbd736b-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402mnvr\" (UID: \"ef55e071-0d79-43e4-b4e1-493b6bbd736b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402mnvr" Jul 23 00:16:23.061913 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:23.061858 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ef55e071-0d79-43e4-b4e1-493b6bbd736b-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402mnvr\" (UID: \"ef55e071-0d79-43e4-b4e1-493b6bbd736b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402mnvr" Jul 23 00:16:23.070586 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:23.070561 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d74r6\" (UniqueName: \"kubernetes.io/projected/ef55e071-0d79-43e4-b4e1-493b6bbd736b-kube-api-access-d74r6\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402mnvr\" (UID: \"ef55e071-0d79-43e4-b4e1-493b6bbd736b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402mnvr" Jul 23 00:16:23.218872 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:23.218788 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402mnvr" Jul 23 00:16:23.361874 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:23.361840 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402mnvr"] Jul 23 00:16:23.364396 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:16:23.364366 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef55e071_0d79_43e4_b4e1_493b6bbd736b.slice/crio-0147202a7033fd96d544ba3f07017182569b6ef1a55ecc8f56dede6db8105017 WatchSource:0}: Error finding container 0147202a7033fd96d544ba3f07017182569b6ef1a55ecc8f56dede6db8105017: Status 404 returned error can't find the container with id 0147202a7033fd96d544ba3f07017182569b6ef1a55ecc8f56dede6db8105017 Jul 23 00:16:23.755865 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:23.755826 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402mnvr" event={"ID":"ef55e071-0d79-43e4-b4e1-493b6bbd736b","Type":"ContainerStarted","Data":"0147202a7033fd96d544ba3f07017182569b6ef1a55ecc8f56dede6db8105017"} Jul 23 00:16:30.787331 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:30.787281 2562 generic.go:356] "Generic (PLEG): container finished" podID="ef55e071-0d79-43e4-b4e1-493b6bbd736b" containerID="d23ad7ffa9f4c5b468f832f787df2a665fa835fbcf392baf7db6541d1bf99836" exitCode=0 Jul 23 00:16:30.787738 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:30.787356 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402mnvr" event={"ID":"ef55e071-0d79-43e4-b4e1-493b6bbd736b","Type":"ContainerDied","Data":"d23ad7ffa9f4c5b468f832f787df2a665fa835fbcf392baf7db6541d1bf99836"} Jul 23 00:16:33.802554 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:33.802512 2562 generic.go:356] "Generic (PLEG): container finished" podID="ef55e071-0d79-43e4-b4e1-493b6bbd736b" containerID="0d13a4ef4c02be25a9069555e7de9d1c3fb3f1e35a0e4cce8654ff8ad1342fab" exitCode=0 Jul 23 00:16:33.802995 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:33.802589 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402mnvr" event={"ID":"ef55e071-0d79-43e4-b4e1-493b6bbd736b","Type":"ContainerDied","Data":"0d13a4ef4c02be25a9069555e7de9d1c3fb3f1e35a0e4cce8654ff8ad1342fab"} Jul 23 00:16:41.839929 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:41.839892 2562 generic.go:356] "Generic (PLEG): container finished" podID="ef55e071-0d79-43e4-b4e1-493b6bbd736b" containerID="696959720c5f0e5969e0ab3e7cc72a4768b049ee22d5544c5af17ea7c19b0387" exitCode=0 Jul 23 00:16:41.840348 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:41.840007 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402mnvr" event={"ID":"ef55e071-0d79-43e4-b4e1-493b6bbd736b","Type":"ContainerDied","Data":"696959720c5f0e5969e0ab3e7cc72a4768b049ee22d5544c5af17ea7c19b0387"} Jul 23 00:16:42.979176 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:42.979149 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402mnvr" Jul 23 00:16:43.138802 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:43.138673 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d74r6\" (UniqueName: \"kubernetes.io/projected/ef55e071-0d79-43e4-b4e1-493b6bbd736b-kube-api-access-d74r6\") pod \"ef55e071-0d79-43e4-b4e1-493b6bbd736b\" (UID: \"ef55e071-0d79-43e4-b4e1-493b6bbd736b\") " Jul 23 00:16:43.138802 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:43.138725 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ef55e071-0d79-43e4-b4e1-493b6bbd736b-util\") pod \"ef55e071-0d79-43e4-b4e1-493b6bbd736b\" (UID: \"ef55e071-0d79-43e4-b4e1-493b6bbd736b\") " Jul 23 00:16:43.138802 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:43.138797 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef55e071-0d79-43e4-b4e1-493b6bbd736b-bundle\") pod \"ef55e071-0d79-43e4-b4e1-493b6bbd736b\" (UID: \"ef55e071-0d79-43e4-b4e1-493b6bbd736b\") " Jul 23 00:16:43.139529 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:43.139501 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef55e071-0d79-43e4-b4e1-493b6bbd736b-bundle" (OuterVolumeSpecName: "bundle") pod "ef55e071-0d79-43e4-b4e1-493b6bbd736b" (UID: "ef55e071-0d79-43e4-b4e1-493b6bbd736b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:16:43.140936 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:43.140912 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef55e071-0d79-43e4-b4e1-493b6bbd736b-kube-api-access-d74r6" (OuterVolumeSpecName: "kube-api-access-d74r6") pod "ef55e071-0d79-43e4-b4e1-493b6bbd736b" (UID: "ef55e071-0d79-43e4-b4e1-493b6bbd736b"). InnerVolumeSpecName "kube-api-access-d74r6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:16:43.144211 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:43.144181 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef55e071-0d79-43e4-b4e1-493b6bbd736b-util" (OuterVolumeSpecName: "util") pod "ef55e071-0d79-43e4-b4e1-493b6bbd736b" (UID: "ef55e071-0d79-43e4-b4e1-493b6bbd736b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:16:43.239712 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:43.239663 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef55e071-0d79-43e4-b4e1-493b6bbd736b-bundle\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:16:43.239712 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:43.239702 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d74r6\" (UniqueName: \"kubernetes.io/projected/ef55e071-0d79-43e4-b4e1-493b6bbd736b-kube-api-access-d74r6\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:16:43.239712 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:43.239715 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ef55e071-0d79-43e4-b4e1-493b6bbd736b-util\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:16:43.850371 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:43.850332 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402mnvr" event={"ID":"ef55e071-0d79-43e4-b4e1-493b6bbd736b","Type":"ContainerDied","Data":"0147202a7033fd96d544ba3f07017182569b6ef1a55ecc8f56dede6db8105017"} Jul 23 00:16:43.850552 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:43.850378 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402mnvr" Jul 23 00:16:43.850552 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:43.850384 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0147202a7033fd96d544ba3f07017182569b6ef1a55ecc8f56dede6db8105017" Jul 23 00:16:50.903065 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:50.903025 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kh6dg"] Jul 23 00:16:50.903485 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:50.903372 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef55e071-0d79-43e4-b4e1-493b6bbd736b" containerName="extract" Jul 23 00:16:50.903485 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:50.903384 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef55e071-0d79-43e4-b4e1-493b6bbd736b" containerName="extract" Jul 23 00:16:50.903485 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:50.903396 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef55e071-0d79-43e4-b4e1-493b6bbd736b" containerName="util" Jul 23 00:16:50.903485 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:50.903401 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef55e071-0d79-43e4-b4e1-493b6bbd736b" containerName="util" Jul 23 00:16:50.903485 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:50.903416 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef55e071-0d79-43e4-b4e1-493b6bbd736b" containerName="pull" Jul 23 00:16:50.903485 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:50.903421 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef55e071-0d79-43e4-b4e1-493b6bbd736b" containerName="pull" Jul 23 00:16:50.903485 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:50.903472 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef55e071-0d79-43e4-b4e1-493b6bbd736b" containerName="extract" Jul 23 00:16:50.907479 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:50.907453 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kh6dg" Jul 23 00:16:50.912866 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:50.912832 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-6xgwx\"" Jul 23 00:16:50.912866 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:50.912853 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 23 00:16:50.912866 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:50.912869 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 23 00:16:50.920166 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:50.920127 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kh6dg"] Jul 23 00:16:51.006918 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:51.006886 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4kbp\" (UniqueName: \"kubernetes.io/projected/f6c0c0b2-0a3e-4134-b107-c892cb32ee0d-kube-api-access-s4kbp\") pod \"cert-manager-operator-controller-manager-55c444bf86-kh6dg\" (UID: \"f6c0c0b2-0a3e-4134-b107-c892cb32ee0d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kh6dg" Jul 23 00:16:51.007091 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:51.006968 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6c0c0b2-0a3e-4134-b107-c892cb32ee0d-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-kh6dg\" (UID: \"f6c0c0b2-0a3e-4134-b107-c892cb32ee0d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kh6dg" Jul 23 00:16:51.107610 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:51.107566 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4kbp\" (UniqueName: \"kubernetes.io/projected/f6c0c0b2-0a3e-4134-b107-c892cb32ee0d-kube-api-access-s4kbp\") pod \"cert-manager-operator-controller-manager-55c444bf86-kh6dg\" (UID: \"f6c0c0b2-0a3e-4134-b107-c892cb32ee0d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kh6dg" Jul 23 00:16:51.107840 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:51.107627 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6c0c0b2-0a3e-4134-b107-c892cb32ee0d-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-kh6dg\" (UID: \"f6c0c0b2-0a3e-4134-b107-c892cb32ee0d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kh6dg" Jul 23 00:16:51.108053 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:51.108030 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6c0c0b2-0a3e-4134-b107-c892cb32ee0d-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-kh6dg\" (UID: \"f6c0c0b2-0a3e-4134-b107-c892cb32ee0d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kh6dg" Jul 23 00:16:51.118542 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:51.118510 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4kbp\" (UniqueName: \"kubernetes.io/projected/f6c0c0b2-0a3e-4134-b107-c892cb32ee0d-kube-api-access-s4kbp\") pod \"cert-manager-operator-controller-manager-55c444bf86-kh6dg\" (UID: \"f6c0c0b2-0a3e-4134-b107-c892cb32ee0d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kh6dg" Jul 23 00:16:51.217013 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:51.216924 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kh6dg" Jul 23 00:16:51.365044 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:51.365017 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kh6dg"] Jul 23 00:16:51.367623 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:16:51.367593 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6c0c0b2_0a3e_4134_b107_c892cb32ee0d.slice/crio-b6cf78ed5607512229a232dc72c9f7fef2f2812c1b7da351b83d8bb28985a4cb WatchSource:0}: Error finding container b6cf78ed5607512229a232dc72c9f7fef2f2812c1b7da351b83d8bb28985a4cb: Status 404 returned error can't find the container with id b6cf78ed5607512229a232dc72c9f7fef2f2812c1b7da351b83d8bb28985a4cb Jul 23 00:16:51.886035 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:51.885996 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kh6dg" event={"ID":"f6c0c0b2-0a3e-4134-b107-c892cb32ee0d","Type":"ContainerStarted","Data":"b6cf78ed5607512229a232dc72c9f7fef2f2812c1b7da351b83d8bb28985a4cb"} Jul 23 00:16:53.898991 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:53.898943 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kh6dg" event={"ID":"f6c0c0b2-0a3e-4134-b107-c892cb32ee0d","Type":"ContainerStarted","Data":"448e0635558dd19a3530428a908e3fd16809fd50d019d0c4258064fe71009d34"} Jul 23 00:16:53.923951 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:53.923244 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kh6dg" podStartSLOduration=2.025459444 podStartE2EDuration="3.923221502s" totalImagesPullingTime="1.897762058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:16:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:16:53.921550938 +0000 UTC m=+527.717057795" watchObservedRunningTime="2026-07-23 00:16:53.923221502 +0000 UTC m=+527.718728360" Jul 23 00:16:54.993775 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:54.993717 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9q7mh"] Jul 23 00:16:54.996233 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:54.996214 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9q7mh" Jul 23 00:16:54.999026 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:54.998991 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 23 00:16:54.999026 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:54.998989 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 23 00:16:54.999244 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:54.999060 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-l2sn7\"" Jul 23 00:16:55.013400 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:55.013358 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9q7mh"] Jul 23 00:16:55.047245 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:55.047208 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa4f7e09-a40f-441c-be02-a155c011edcb-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9q7mh\" (UID: \"aa4f7e09-a40f-441c-be02-a155c011edcb\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9q7mh" Jul 23 00:16:55.047245 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:55.047252 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa4f7e09-a40f-441c-be02-a155c011edcb-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9q7mh\" (UID: \"aa4f7e09-a40f-441c-be02-a155c011edcb\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9q7mh" Jul 23 00:16:55.047531 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:55.047274 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-567zq\" (UniqueName: \"kubernetes.io/projected/aa4f7e09-a40f-441c-be02-a155c011edcb-kube-api-access-567zq\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9q7mh\" (UID: \"aa4f7e09-a40f-441c-be02-a155c011edcb\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9q7mh" Jul 23 00:16:55.148182 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:55.148138 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa4f7e09-a40f-441c-be02-a155c011edcb-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9q7mh\" (UID: \"aa4f7e09-a40f-441c-be02-a155c011edcb\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9q7mh" Jul 23 00:16:55.148182 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:55.148179 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa4f7e09-a40f-441c-be02-a155c011edcb-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9q7mh\" (UID: \"aa4f7e09-a40f-441c-be02-a155c011edcb\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9q7mh" Jul 23 00:16:55.148397 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:55.148200 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-567zq\" (UniqueName: \"kubernetes.io/projected/aa4f7e09-a40f-441c-be02-a155c011edcb-kube-api-access-567zq\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9q7mh\" (UID: \"aa4f7e09-a40f-441c-be02-a155c011edcb\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9q7mh" Jul 23 00:16:55.148596 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:55.148576 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa4f7e09-a40f-441c-be02-a155c011edcb-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9q7mh\" (UID: \"aa4f7e09-a40f-441c-be02-a155c011edcb\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9q7mh" Jul 23 00:16:55.148632 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:55.148595 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa4f7e09-a40f-441c-be02-a155c011edcb-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9q7mh\" (UID: \"aa4f7e09-a40f-441c-be02-a155c011edcb\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9q7mh" Jul 23 00:16:55.159635 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:55.159602 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-567zq\" (UniqueName: \"kubernetes.io/projected/aa4f7e09-a40f-441c-be02-a155c011edcb-kube-api-access-567zq\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9q7mh\" (UID: \"aa4f7e09-a40f-441c-be02-a155c011edcb\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9q7mh" Jul 23 00:16:55.305876 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:55.305835 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9q7mh" Jul 23 00:16:55.458013 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:55.457972 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9q7mh"] Jul 23 00:16:55.461043 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:16:55.461014 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa4f7e09_a40f_441c_be02_a155c011edcb.slice/crio-d5b74e0117ed353d4f6b71b85646b5f29c5c374e0deccf229eccd4189eb2454a WatchSource:0}: Error finding container d5b74e0117ed353d4f6b71b85646b5f29c5c374e0deccf229eccd4189eb2454a: Status 404 returned error can't find the container with id d5b74e0117ed353d4f6b71b85646b5f29c5c374e0deccf229eccd4189eb2454a Jul 23 00:16:55.910084 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:55.909982 2562 generic.go:356] "Generic (PLEG): container finished" podID="aa4f7e09-a40f-441c-be02-a155c011edcb" containerID="6e8f750c37ede3c0fc0031e45d16fad4c28a258e250b1016101963f1530ae938" exitCode=0 Jul 23 00:16:55.910242 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:55.910083 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9q7mh" event={"ID":"aa4f7e09-a40f-441c-be02-a155c011edcb","Type":"ContainerDied","Data":"6e8f750c37ede3c0fc0031e45d16fad4c28a258e250b1016101963f1530ae938"} Jul 23 00:16:55.910242 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:55.910156 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9q7mh" event={"ID":"aa4f7e09-a40f-441c-be02-a155c011edcb","Type":"ContainerStarted","Data":"d5b74e0117ed353d4f6b71b85646b5f29c5c374e0deccf229eccd4189eb2454a"} Jul 23 00:16:58.925795 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:58.925727 2562 generic.go:356] "Generic (PLEG): container finished" podID="aa4f7e09-a40f-441c-be02-a155c011edcb" containerID="9fa3b7fe35b2149e55d9775516ffa27aba33d1285c82bb2baded5dc9b5f86af2" exitCode=0 Jul 23 00:16:58.926202 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:58.925805 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9q7mh" event={"ID":"aa4f7e09-a40f-441c-be02-a155c011edcb","Type":"ContainerDied","Data":"9fa3b7fe35b2149e55d9775516ffa27aba33d1285c82bb2baded5dc9b5f86af2"} Jul 23 00:16:59.932872 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:59.932831 2562 generic.go:356] "Generic (PLEG): container finished" podID="aa4f7e09-a40f-441c-be02-a155c011edcb" containerID="4662ebd714d893177fac733f0afd48039c071bcf17a05740a6dd1a308c86d6f7" exitCode=0 Jul 23 00:16:59.933293 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:16:59.932908 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9q7mh" event={"ID":"aa4f7e09-a40f-441c-be02-a155c011edcb","Type":"ContainerDied","Data":"4662ebd714d893177fac733f0afd48039c071bcf17a05740a6dd1a308c86d6f7"} Jul 23 00:17:01.069626 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:01.069601 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9q7mh" Jul 23 00:17:01.101367 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:01.101330 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa4f7e09-a40f-441c-be02-a155c011edcb-bundle\") pod \"aa4f7e09-a40f-441c-be02-a155c011edcb\" (UID: \"aa4f7e09-a40f-441c-be02-a155c011edcb\") " Jul 23 00:17:01.101574 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:01.101382 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-567zq\" (UniqueName: \"kubernetes.io/projected/aa4f7e09-a40f-441c-be02-a155c011edcb-kube-api-access-567zq\") pod \"aa4f7e09-a40f-441c-be02-a155c011edcb\" (UID: \"aa4f7e09-a40f-441c-be02-a155c011edcb\") " Jul 23 00:17:01.101574 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:01.101448 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa4f7e09-a40f-441c-be02-a155c011edcb-util\") pod \"aa4f7e09-a40f-441c-be02-a155c011edcb\" (UID: \"aa4f7e09-a40f-441c-be02-a155c011edcb\") " Jul 23 00:17:01.101830 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:01.101807 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa4f7e09-a40f-441c-be02-a155c011edcb-bundle" (OuterVolumeSpecName: "bundle") pod "aa4f7e09-a40f-441c-be02-a155c011edcb" (UID: "aa4f7e09-a40f-441c-be02-a155c011edcb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:17:01.103556 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:01.103529 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa4f7e09-a40f-441c-be02-a155c011edcb-kube-api-access-567zq" (OuterVolumeSpecName: "kube-api-access-567zq") pod "aa4f7e09-a40f-441c-be02-a155c011edcb" (UID: "aa4f7e09-a40f-441c-be02-a155c011edcb"). InnerVolumeSpecName "kube-api-access-567zq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:17:01.131857 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:01.131810 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa4f7e09-a40f-441c-be02-a155c011edcb-util" (OuterVolumeSpecName: "util") pod "aa4f7e09-a40f-441c-be02-a155c011edcb" (UID: "aa4f7e09-a40f-441c-be02-a155c011edcb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:17:01.202190 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:01.202090 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa4f7e09-a40f-441c-be02-a155c011edcb-util\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:17:01.202190 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:01.202125 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa4f7e09-a40f-441c-be02-a155c011edcb-bundle\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:17:01.202190 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:01.202138 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-567zq\" (UniqueName: \"kubernetes.io/projected/aa4f7e09-a40f-441c-be02-a155c011edcb-kube-api-access-567zq\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:17:01.943663 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:01.943631 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9q7mh" event={"ID":"aa4f7e09-a40f-441c-be02-a155c011edcb","Type":"ContainerDied","Data":"d5b74e0117ed353d4f6b71b85646b5f29c5c374e0deccf229eccd4189eb2454a"} Jul 23 00:17:01.943858 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:01.943680 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5b74e0117ed353d4f6b71b85646b5f29c5c374e0deccf229eccd4189eb2454a" Jul 23 00:17:01.943858 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:01.943679 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9q7mh" Jul 23 00:17:07.590749 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:07.590708 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9qwh7"] Jul 23 00:17:07.591228 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:07.591056 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa4f7e09-a40f-441c-be02-a155c011edcb" containerName="pull" Jul 23 00:17:07.591228 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:07.591067 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa4f7e09-a40f-441c-be02-a155c011edcb" containerName="pull" Jul 23 00:17:07.591228 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:07.591077 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa4f7e09-a40f-441c-be02-a155c011edcb" containerName="extract" Jul 23 00:17:07.591228 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:07.591083 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa4f7e09-a40f-441c-be02-a155c011edcb" containerName="extract" Jul 23 00:17:07.591228 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:07.591097 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa4f7e09-a40f-441c-be02-a155c011edcb" containerName="util" Jul 23 00:17:07.591228 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:07.591103 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa4f7e09-a40f-441c-be02-a155c011edcb" containerName="util" Jul 23 00:17:07.591228 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:07.591159 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa4f7e09-a40f-441c-be02-a155c011edcb" containerName="extract" Jul 23 00:17:07.594276 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:07.594257 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9qwh7" Jul 23 00:17:07.596975 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:07.596951 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-zf677\"" Jul 23 00:17:07.597082 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:07.596980 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 23 00:17:07.598080 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:07.598066 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 23 00:17:07.602619 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:07.602593 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9qwh7"] Jul 23 00:17:07.660277 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:07.660231 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1dc803ca-ac1a-441e-8d4f-d146e45ac96a-tmp\") pod \"openshift-lws-operator-fd8ccff4c-9qwh7\" (UID: \"1dc803ca-ac1a-441e-8d4f-d146e45ac96a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9qwh7" Jul 23 00:17:07.660277 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:07.660280 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7h7j\" (UniqueName: \"kubernetes.io/projected/1dc803ca-ac1a-441e-8d4f-d146e45ac96a-kube-api-access-q7h7j\") pod \"openshift-lws-operator-fd8ccff4c-9qwh7\" (UID: \"1dc803ca-ac1a-441e-8d4f-d146e45ac96a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9qwh7" Jul 23 00:17:07.761625 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:07.761583 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1dc803ca-ac1a-441e-8d4f-d146e45ac96a-tmp\") pod \"openshift-lws-operator-fd8ccff4c-9qwh7\" (UID: \"1dc803ca-ac1a-441e-8d4f-d146e45ac96a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9qwh7" Jul 23 00:17:07.761625 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:07.761630 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7h7j\" (UniqueName: \"kubernetes.io/projected/1dc803ca-ac1a-441e-8d4f-d146e45ac96a-kube-api-access-q7h7j\") pod \"openshift-lws-operator-fd8ccff4c-9qwh7\" (UID: \"1dc803ca-ac1a-441e-8d4f-d146e45ac96a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9qwh7" Jul 23 00:17:07.762027 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:07.762005 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1dc803ca-ac1a-441e-8d4f-d146e45ac96a-tmp\") pod \"openshift-lws-operator-fd8ccff4c-9qwh7\" (UID: \"1dc803ca-ac1a-441e-8d4f-d146e45ac96a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9qwh7" Jul 23 00:17:07.774591 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:07.774559 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7h7j\" (UniqueName: \"kubernetes.io/projected/1dc803ca-ac1a-441e-8d4f-d146e45ac96a-kube-api-access-q7h7j\") pod \"openshift-lws-operator-fd8ccff4c-9qwh7\" (UID: \"1dc803ca-ac1a-441e-8d4f-d146e45ac96a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9qwh7" Jul 23 00:17:07.915935 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:07.915848 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9qwh7" Jul 23 00:17:08.056019 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:08.055971 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9qwh7"] Jul 23 00:17:08.058731 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:17:08.058703 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1dc803ca_ac1a_441e_8d4f_d146e45ac96a.slice/crio-29d7a856e9f3b763c18d98847ff24f107d80181aae6597821ff58dc7d25c0a10 WatchSource:0}: Error finding container 29d7a856e9f3b763c18d98847ff24f107d80181aae6597821ff58dc7d25c0a10: Status 404 returned error can't find the container with id 29d7a856e9f3b763c18d98847ff24f107d80181aae6597821ff58dc7d25c0a10 Jul 23 00:17:08.981384 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:08.981344 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9qwh7" event={"ID":"1dc803ca-ac1a-441e-8d4f-d146e45ac96a","Type":"ContainerStarted","Data":"29d7a856e9f3b763c18d98847ff24f107d80181aae6597821ff58dc7d25c0a10"} Jul 23 00:17:10.992689 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:10.992653 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9qwh7" event={"ID":"1dc803ca-ac1a-441e-8d4f-d146e45ac96a","Type":"ContainerStarted","Data":"3443e02d3e9db5cdb374bd750b718a6a70f5e6875ff44455ccf7d75b2582bdaf"} Jul 23 00:17:11.009682 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:11.009625 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9qwh7" podStartSLOduration=1.624596689 podStartE2EDuration="4.009609267s" totalImagesPullingTime="2.385012578s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:17:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:17:11.008644368 +0000 UTC m=+544.804151226" watchObservedRunningTime="2026-07-23 00:17:11.009609267 +0000 UTC m=+544.805116124" Jul 23 00:17:15.428084 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:15.428037 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmgfkg"] Jul 23 00:17:15.430918 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:15.430892 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmgfkg" Jul 23 00:17:15.433647 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:15.433622 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 23 00:17:15.433819 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:15.433625 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 23 00:17:15.434876 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:15.434861 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-l2sn7\"" Jul 23 00:17:15.439740 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:15.439713 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmgfkg"] Jul 23 00:17:15.532326 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:15.532283 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9e7acc2a-0ee1-4263-84c6-5e609d4ef2fb-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmgfkg\" (UID: \"9e7acc2a-0ee1-4263-84c6-5e609d4ef2fb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmgfkg" Jul 23 00:17:15.532507 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:15.532349 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl7s6\" (UniqueName: \"kubernetes.io/projected/9e7acc2a-0ee1-4263-84c6-5e609d4ef2fb-kube-api-access-zl7s6\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmgfkg\" (UID: \"9e7acc2a-0ee1-4263-84c6-5e609d4ef2fb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmgfkg" Jul 23 00:17:15.532507 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:15.532404 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9e7acc2a-0ee1-4263-84c6-5e609d4ef2fb-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmgfkg\" (UID: \"9e7acc2a-0ee1-4263-84c6-5e609d4ef2fb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmgfkg" Jul 23 00:17:15.633553 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:15.633514 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9e7acc2a-0ee1-4263-84c6-5e609d4ef2fb-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmgfkg\" (UID: \"9e7acc2a-0ee1-4263-84c6-5e609d4ef2fb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmgfkg" Jul 23 00:17:15.633785 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:15.633571 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9e7acc2a-0ee1-4263-84c6-5e609d4ef2fb-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmgfkg\" (UID: \"9e7acc2a-0ee1-4263-84c6-5e609d4ef2fb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmgfkg" Jul 23 00:17:15.633785 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:15.633647 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl7s6\" (UniqueName: \"kubernetes.io/projected/9e7acc2a-0ee1-4263-84c6-5e609d4ef2fb-kube-api-access-zl7s6\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmgfkg\" (UID: \"9e7acc2a-0ee1-4263-84c6-5e609d4ef2fb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmgfkg" Jul 23 00:17:15.634031 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:15.634011 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9e7acc2a-0ee1-4263-84c6-5e609d4ef2fb-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmgfkg\" (UID: \"9e7acc2a-0ee1-4263-84c6-5e609d4ef2fb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmgfkg" Jul 23 00:17:15.634071 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:15.634010 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9e7acc2a-0ee1-4263-84c6-5e609d4ef2fb-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmgfkg\" (UID: \"9e7acc2a-0ee1-4263-84c6-5e609d4ef2fb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmgfkg" Jul 23 00:17:15.642140 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:15.642108 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl7s6\" (UniqueName: \"kubernetes.io/projected/9e7acc2a-0ee1-4263-84c6-5e609d4ef2fb-kube-api-access-zl7s6\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmgfkg\" (UID: \"9e7acc2a-0ee1-4263-84c6-5e609d4ef2fb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmgfkg" Jul 23 00:17:15.741455 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:15.741355 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmgfkg" Jul 23 00:17:15.878545 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:15.878517 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmgfkg"] Jul 23 00:17:15.880780 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:17:15.880735 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e7acc2a_0ee1_4263_84c6_5e609d4ef2fb.slice/crio-c16aa3f740bc73b5e3a0a9a065348a988f791045e6cbfd84418d31db625633cd WatchSource:0}: Error finding container c16aa3f740bc73b5e3a0a9a065348a988f791045e6cbfd84418d31db625633cd: Status 404 returned error can't find the container with id c16aa3f740bc73b5e3a0a9a065348a988f791045e6cbfd84418d31db625633cd Jul 23 00:17:16.016972 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:16.016887 2562 generic.go:356] "Generic (PLEG): container finished" podID="9e7acc2a-0ee1-4263-84c6-5e609d4ef2fb" containerID="52fe20902ae8508e50503965180257556207d02c83c4edc78fb86c7133342b65" exitCode=0 Jul 23 00:17:16.016972 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:16.016931 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmgfkg" event={"ID":"9e7acc2a-0ee1-4263-84c6-5e609d4ef2fb","Type":"ContainerDied","Data":"52fe20902ae8508e50503965180257556207d02c83c4edc78fb86c7133342b65"} Jul 23 00:17:16.016972 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:16.016967 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmgfkg" event={"ID":"9e7acc2a-0ee1-4263-84c6-5e609d4ef2fb","Type":"ContainerStarted","Data":"c16aa3f740bc73b5e3a0a9a065348a988f791045e6cbfd84418d31db625633cd"} Jul 23 00:17:17.023492 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:17.023454 2562 generic.go:356] "Generic (PLEG): container finished" podID="9e7acc2a-0ee1-4263-84c6-5e609d4ef2fb" containerID="eddcf897979f1082a4362999511360cc8eb21f7bd8c30f7be58161d41ddbc42d" exitCode=0 Jul 23 00:17:17.023913 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:17.023536 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmgfkg" event={"ID":"9e7acc2a-0ee1-4263-84c6-5e609d4ef2fb","Type":"ContainerDied","Data":"eddcf897979f1082a4362999511360cc8eb21f7bd8c30f7be58161d41ddbc42d"} Jul 23 00:17:18.030444 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:18.030408 2562 generic.go:356] "Generic (PLEG): container finished" podID="9e7acc2a-0ee1-4263-84c6-5e609d4ef2fb" containerID="5aed8dcc7189b2774725081725d5cec301701e7f53f600dbaa2223263db01fca" exitCode=0 Jul 23 00:17:18.030952 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:18.030489 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmgfkg" event={"ID":"9e7acc2a-0ee1-4263-84c6-5e609d4ef2fb","Type":"ContainerDied","Data":"5aed8dcc7189b2774725081725d5cec301701e7f53f600dbaa2223263db01fca"} Jul 23 00:17:19.170273 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:19.170239 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmgfkg" Jul 23 00:17:19.268008 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:19.267960 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zl7s6\" (UniqueName: \"kubernetes.io/projected/9e7acc2a-0ee1-4263-84c6-5e609d4ef2fb-kube-api-access-zl7s6\") pod \"9e7acc2a-0ee1-4263-84c6-5e609d4ef2fb\" (UID: \"9e7acc2a-0ee1-4263-84c6-5e609d4ef2fb\") " Jul 23 00:17:19.268008 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:19.268014 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9e7acc2a-0ee1-4263-84c6-5e609d4ef2fb-util\") pod \"9e7acc2a-0ee1-4263-84c6-5e609d4ef2fb\" (UID: \"9e7acc2a-0ee1-4263-84c6-5e609d4ef2fb\") " Jul 23 00:17:19.268263 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:19.268137 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9e7acc2a-0ee1-4263-84c6-5e609d4ef2fb-bundle\") pod \"9e7acc2a-0ee1-4263-84c6-5e609d4ef2fb\" (UID: \"9e7acc2a-0ee1-4263-84c6-5e609d4ef2fb\") " Jul 23 00:17:19.269019 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:19.268981 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e7acc2a-0ee1-4263-84c6-5e609d4ef2fb-bundle" (OuterVolumeSpecName: "bundle") pod "9e7acc2a-0ee1-4263-84c6-5e609d4ef2fb" (UID: "9e7acc2a-0ee1-4263-84c6-5e609d4ef2fb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:17:19.270156 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:19.270127 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e7acc2a-0ee1-4263-84c6-5e609d4ef2fb-kube-api-access-zl7s6" (OuterVolumeSpecName: "kube-api-access-zl7s6") pod "9e7acc2a-0ee1-4263-84c6-5e609d4ef2fb" (UID: "9e7acc2a-0ee1-4263-84c6-5e609d4ef2fb"). InnerVolumeSpecName "kube-api-access-zl7s6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:17:19.273934 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:19.273891 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e7acc2a-0ee1-4263-84c6-5e609d4ef2fb-util" (OuterVolumeSpecName: "util") pod "9e7acc2a-0ee1-4263-84c6-5e609d4ef2fb" (UID: "9e7acc2a-0ee1-4263-84c6-5e609d4ef2fb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:17:19.369102 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:19.368996 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9e7acc2a-0ee1-4263-84c6-5e609d4ef2fb-bundle\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:17:19.369102 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:19.369045 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zl7s6\" (UniqueName: \"kubernetes.io/projected/9e7acc2a-0ee1-4263-84c6-5e609d4ef2fb-kube-api-access-zl7s6\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:17:19.369102 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:19.369055 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9e7acc2a-0ee1-4263-84c6-5e609d4ef2fb-util\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:17:20.042349 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:20.042317 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmgfkg" Jul 23 00:17:20.042349 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:20.042346 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmgfkg" event={"ID":"9e7acc2a-0ee1-4263-84c6-5e609d4ef2fb","Type":"ContainerDied","Data":"c16aa3f740bc73b5e3a0a9a065348a988f791045e6cbfd84418d31db625633cd"} Jul 23 00:17:20.042562 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:20.042392 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c16aa3f740bc73b5e3a0a9a065348a988f791045e6cbfd84418d31db625633cd" Jul 23 00:17:27.359873 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:27.359832 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-k8wsd"] Jul 23 00:17:27.361237 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:27.361203 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e7acc2a-0ee1-4263-84c6-5e609d4ef2fb" containerName="util" Jul 23 00:17:27.361395 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:27.361384 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e7acc2a-0ee1-4263-84c6-5e609d4ef2fb" containerName="util" Jul 23 00:17:27.361488 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:27.361478 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e7acc2a-0ee1-4263-84c6-5e609d4ef2fb" containerName="pull" Jul 23 00:17:27.361553 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:27.361545 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e7acc2a-0ee1-4263-84c6-5e609d4ef2fb" containerName="pull" Jul 23 00:17:27.361631 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:27.361622 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e7acc2a-0ee1-4263-84c6-5e609d4ef2fb" containerName="extract" Jul 23 00:17:27.361693 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:27.361685 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e7acc2a-0ee1-4263-84c6-5e609d4ef2fb" containerName="extract" Jul 23 00:17:27.362013 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:27.361996 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e7acc2a-0ee1-4263-84c6-5e609d4ef2fb" containerName="extract" Jul 23 00:17:27.365624 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:27.365594 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-k8wsd" Jul 23 00:17:27.368858 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:27.368832 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 23 00:17:27.368972 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:27.368838 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 23 00:17:27.369089 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:27.369068 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 23 00:17:27.369819 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:27.369804 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 23 00:17:27.372813 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:27.372787 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-mblxv\"" Jul 23 00:17:27.376052 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:27.376024 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-k8wsd"] Jul 23 00:17:27.543410 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:27.543370 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48c0a1e1-c11f-405e-baeb-98d98897dd11-tmp\") pod \"opendatahub-operator-controller-manager-6f65bfdcb7-k8wsd\" (UID: \"48c0a1e1-c11f-405e-baeb-98d98897dd11\") " pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-k8wsd" Jul 23 00:17:27.543598 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:27.543418 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/48c0a1e1-c11f-405e-baeb-98d98897dd11-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f65bfdcb7-k8wsd\" (UID: \"48c0a1e1-c11f-405e-baeb-98d98897dd11\") " pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-k8wsd" Jul 23 00:17:27.543598 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:27.543486 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/48c0a1e1-c11f-405e-baeb-98d98897dd11-manifests\") pod \"opendatahub-operator-controller-manager-6f65bfdcb7-k8wsd\" (UID: \"48c0a1e1-c11f-405e-baeb-98d98897dd11\") " pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-k8wsd" Jul 23 00:17:27.543598 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:27.543509 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/48c0a1e1-c11f-405e-baeb-98d98897dd11-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f65bfdcb7-k8wsd\" (UID: \"48c0a1e1-c11f-405e-baeb-98d98897dd11\") " pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-k8wsd" Jul 23 00:17:27.543598 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:27.543528 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c9zw\" (UniqueName: \"kubernetes.io/projected/48c0a1e1-c11f-405e-baeb-98d98897dd11-kube-api-access-9c9zw\") pod \"opendatahub-operator-controller-manager-6f65bfdcb7-k8wsd\" (UID: \"48c0a1e1-c11f-405e-baeb-98d98897dd11\") " pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-k8wsd" Jul 23 00:17:27.644255 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:27.644154 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48c0a1e1-c11f-405e-baeb-98d98897dd11-tmp\") pod \"opendatahub-operator-controller-manager-6f65bfdcb7-k8wsd\" (UID: \"48c0a1e1-c11f-405e-baeb-98d98897dd11\") " pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-k8wsd" Jul 23 00:17:27.644255 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:27.644196 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/48c0a1e1-c11f-405e-baeb-98d98897dd11-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f65bfdcb7-k8wsd\" (UID: \"48c0a1e1-c11f-405e-baeb-98d98897dd11\") " pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-k8wsd" Jul 23 00:17:27.644255 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:27.644226 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/48c0a1e1-c11f-405e-baeb-98d98897dd11-manifests\") pod \"opendatahub-operator-controller-manager-6f65bfdcb7-k8wsd\" (UID: \"48c0a1e1-c11f-405e-baeb-98d98897dd11\") " pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-k8wsd" Jul 23 00:17:27.644255 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:27.644241 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/48c0a1e1-c11f-405e-baeb-98d98897dd11-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f65bfdcb7-k8wsd\" (UID: \"48c0a1e1-c11f-405e-baeb-98d98897dd11\") " pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-k8wsd" Jul 23 00:17:27.644255 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:27.644256 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c9zw\" (UniqueName: \"kubernetes.io/projected/48c0a1e1-c11f-405e-baeb-98d98897dd11-kube-api-access-9c9zw\") pod \"opendatahub-operator-controller-manager-6f65bfdcb7-k8wsd\" (UID: \"48c0a1e1-c11f-405e-baeb-98d98897dd11\") " pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-k8wsd" Jul 23 00:17:27.644667 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:27.644640 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/48c0a1e1-c11f-405e-baeb-98d98897dd11-manifests\") pod \"opendatahub-operator-controller-manager-6f65bfdcb7-k8wsd\" (UID: \"48c0a1e1-c11f-405e-baeb-98d98897dd11\") " pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-k8wsd" Jul 23 00:17:27.646618 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:27.646594 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48c0a1e1-c11f-405e-baeb-98d98897dd11-tmp\") pod \"opendatahub-operator-controller-manager-6f65bfdcb7-k8wsd\" (UID: \"48c0a1e1-c11f-405e-baeb-98d98897dd11\") " pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-k8wsd" Jul 23 00:17:27.646735 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:27.646656 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/48c0a1e1-c11f-405e-baeb-98d98897dd11-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f65bfdcb7-k8wsd\" (UID: \"48c0a1e1-c11f-405e-baeb-98d98897dd11\") " pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-k8wsd" Jul 23 00:17:27.646879 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:27.646860 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/48c0a1e1-c11f-405e-baeb-98d98897dd11-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f65bfdcb7-k8wsd\" (UID: \"48c0a1e1-c11f-405e-baeb-98d98897dd11\") " pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-k8wsd" Jul 23 00:17:27.655734 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:27.655707 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c9zw\" (UniqueName: \"kubernetes.io/projected/48c0a1e1-c11f-405e-baeb-98d98897dd11-kube-api-access-9c9zw\") pod \"opendatahub-operator-controller-manager-6f65bfdcb7-k8wsd\" (UID: \"48c0a1e1-c11f-405e-baeb-98d98897dd11\") " pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-k8wsd" Jul 23 00:17:27.677470 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:27.677429 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-k8wsd" Jul 23 00:17:27.834909 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:27.834879 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-k8wsd"] Jul 23 00:17:27.837520 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:17:27.837492 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48c0a1e1_c11f_405e_baeb_98d98897dd11.slice/crio-c85b64b547305962efed87b6218cb32fc1e2efbde088f3bceb6ef03931c18041 WatchSource:0}: Error finding container c85b64b547305962efed87b6218cb32fc1e2efbde088f3bceb6ef03931c18041: Status 404 returned error can't find the container with id c85b64b547305962efed87b6218cb32fc1e2efbde088f3bceb6ef03931c18041 Jul 23 00:17:28.077574 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:28.077540 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-k8wsd" event={"ID":"48c0a1e1-c11f-405e-baeb-98d98897dd11","Type":"ContainerStarted","Data":"c85b64b547305962efed87b6218cb32fc1e2efbde088f3bceb6ef03931c18041"} Jul 23 00:17:31.095492 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:31.095398 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-k8wsd" event={"ID":"48c0a1e1-c11f-405e-baeb-98d98897dd11","Type":"ContainerStarted","Data":"f3d9218e7cdf8b9d1395e9db959e7a2ec2fa4507ed604b7405a1b4ac3d85ee4f"} Jul 23 00:17:32.102055 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:32.102017 2562 generic.go:356] "Generic (PLEG): container finished" podID="48c0a1e1-c11f-405e-baeb-98d98897dd11" containerID="f3d9218e7cdf8b9d1395e9db959e7a2ec2fa4507ed604b7405a1b4ac3d85ee4f" exitCode=0 Jul 23 00:17:32.102055 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:32.102062 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-k8wsd" event={"ID":"48c0a1e1-c11f-405e-baeb-98d98897dd11","Type":"ContainerDied","Data":"f3d9218e7cdf8b9d1395e9db959e7a2ec2fa4507ed604b7405a1b4ac3d85ee4f"} Jul 23 00:17:33.108945 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:33.108911 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-k8wsd" event={"ID":"48c0a1e1-c11f-405e-baeb-98d98897dd11","Type":"ContainerStarted","Data":"57421ca86f7cf0bb915842c112562433b2fcbb8aae3b314b4f2f8e8b2d822274"} Jul 23 00:17:33.109409 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:33.109000 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-k8wsd" Jul 23 00:17:33.132958 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:33.132896 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-k8wsd" podStartSLOduration=1.558595479 podStartE2EDuration="6.13287846s" totalImagesPullingTime="3.478842253s" totalInitContainerRuntime="1.095440728s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:17:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:17:33.130409149 +0000 UTC m=+566.925916008" watchObservedRunningTime="2026-07-23 00:17:33.13287846 +0000 UTC m=+566.928385319" Jul 23 00:17:44.116713 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:44.116677 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-k8wsd" Jul 23 00:17:46.611301 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:46.611259 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-85d467fcb7-bn4gl"] Jul 23 00:17:46.622789 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:46.622740 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-85d467fcb7-bn4gl" Jul 23 00:17:46.632533 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:46.632507 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-g4bpr\"" Jul 23 00:17:46.632694 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:46.632510 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 23 00:17:46.632694 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:46.632545 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 23 00:17:46.633094 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:46.633060 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 23 00:17:46.651782 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:46.651715 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-85d467fcb7-bn4gl"] Jul 23 00:17:46.701068 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:46.701024 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a17498fa-9c38-4f7c-958a-bb029dfdabac-cert\") pod \"lws-controller-manager-85d467fcb7-bn4gl\" (UID: \"a17498fa-9c38-4f7c-958a-bb029dfdabac\") " pod="openshift-lws-operator/lws-controller-manager-85d467fcb7-bn4gl" Jul 23 00:17:46.701274 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:46.701105 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4lck\" (UniqueName: \"kubernetes.io/projected/a17498fa-9c38-4f7c-958a-bb029dfdabac-kube-api-access-q4lck\") pod \"lws-controller-manager-85d467fcb7-bn4gl\" (UID: \"a17498fa-9c38-4f7c-958a-bb029dfdabac\") " pod="openshift-lws-operator/lws-controller-manager-85d467fcb7-bn4gl" Jul 23 00:17:46.701274 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:46.701144 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a17498fa-9c38-4f7c-958a-bb029dfdabac-manager-config\") pod \"lws-controller-manager-85d467fcb7-bn4gl\" (UID: \"a17498fa-9c38-4f7c-958a-bb029dfdabac\") " pod="openshift-lws-operator/lws-controller-manager-85d467fcb7-bn4gl" Jul 23 00:17:46.701274 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:46.701165 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a17498fa-9c38-4f7c-958a-bb029dfdabac-metrics-cert\") pod \"lws-controller-manager-85d467fcb7-bn4gl\" (UID: \"a17498fa-9c38-4f7c-958a-bb029dfdabac\") " pod="openshift-lws-operator/lws-controller-manager-85d467fcb7-bn4gl" Jul 23 00:17:46.801695 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:46.801650 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a17498fa-9c38-4f7c-958a-bb029dfdabac-metrics-cert\") pod \"lws-controller-manager-85d467fcb7-bn4gl\" (UID: \"a17498fa-9c38-4f7c-958a-bb029dfdabac\") " pod="openshift-lws-operator/lws-controller-manager-85d467fcb7-bn4gl" Jul 23 00:17:46.801906 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:46.801768 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a17498fa-9c38-4f7c-958a-bb029dfdabac-cert\") pod \"lws-controller-manager-85d467fcb7-bn4gl\" (UID: \"a17498fa-9c38-4f7c-958a-bb029dfdabac\") " pod="openshift-lws-operator/lws-controller-manager-85d467fcb7-bn4gl" Jul 23 00:17:46.801906 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:46.801844 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4lck\" (UniqueName: \"kubernetes.io/projected/a17498fa-9c38-4f7c-958a-bb029dfdabac-kube-api-access-q4lck\") pod \"lws-controller-manager-85d467fcb7-bn4gl\" (UID: \"a17498fa-9c38-4f7c-958a-bb029dfdabac\") " pod="openshift-lws-operator/lws-controller-manager-85d467fcb7-bn4gl" Jul 23 00:17:46.801906 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:46.801890 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a17498fa-9c38-4f7c-958a-bb029dfdabac-manager-config\") pod \"lws-controller-manager-85d467fcb7-bn4gl\" (UID: \"a17498fa-9c38-4f7c-958a-bb029dfdabac\") " pod="openshift-lws-operator/lws-controller-manager-85d467fcb7-bn4gl" Jul 23 00:17:46.802623 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:46.802595 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a17498fa-9c38-4f7c-958a-bb029dfdabac-manager-config\") pod \"lws-controller-manager-85d467fcb7-bn4gl\" (UID: \"a17498fa-9c38-4f7c-958a-bb029dfdabac\") " pod="openshift-lws-operator/lws-controller-manager-85d467fcb7-bn4gl" Jul 23 00:17:46.804551 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:46.804518 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a17498fa-9c38-4f7c-958a-bb029dfdabac-cert\") pod \"lws-controller-manager-85d467fcb7-bn4gl\" (UID: \"a17498fa-9c38-4f7c-958a-bb029dfdabac\") " pod="openshift-lws-operator/lws-controller-manager-85d467fcb7-bn4gl" Jul 23 00:17:46.804676 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:46.804550 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a17498fa-9c38-4f7c-958a-bb029dfdabac-metrics-cert\") pod \"lws-controller-manager-85d467fcb7-bn4gl\" (UID: \"a17498fa-9c38-4f7c-958a-bb029dfdabac\") " pod="openshift-lws-operator/lws-controller-manager-85d467fcb7-bn4gl" Jul 23 00:17:46.817125 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:46.817086 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4lck\" (UniqueName: \"kubernetes.io/projected/a17498fa-9c38-4f7c-958a-bb029dfdabac-kube-api-access-q4lck\") pod \"lws-controller-manager-85d467fcb7-bn4gl\" (UID: \"a17498fa-9c38-4f7c-958a-bb029dfdabac\") " pod="openshift-lws-operator/lws-controller-manager-85d467fcb7-bn4gl" Jul 23 00:17:46.933229 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:46.933130 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-85d467fcb7-bn4gl" Jul 23 00:17:47.096671 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:47.096643 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-85d467fcb7-bn4gl"] Jul 23 00:17:47.099197 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:17:47.099163 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda17498fa_9c38_4f7c_958a_bb029dfdabac.slice/crio-78eb99be6608714aa7efb99e49b1aaab9923fc063463b9908d08586566c7e78c WatchSource:0}: Error finding container 78eb99be6608714aa7efb99e49b1aaab9923fc063463b9908d08586566c7e78c: Status 404 returned error can't find the container with id 78eb99be6608714aa7efb99e49b1aaab9923fc063463b9908d08586566c7e78c Jul 23 00:17:47.180033 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:47.179988 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-85d467fcb7-bn4gl" event={"ID":"a17498fa-9c38-4f7c-958a-bb029dfdabac","Type":"ContainerStarted","Data":"78eb99be6608714aa7efb99e49b1aaab9923fc063463b9908d08586566c7e78c"} Jul 23 00:17:49.191404 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:49.191363 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-85d467fcb7-bn4gl" event={"ID":"a17498fa-9c38-4f7c-958a-bb029dfdabac","Type":"ContainerStarted","Data":"93b62e722c7dddbe587a58d930260648007dca613c7e09d4dc7267b9926d698f"} Jul 23 00:17:49.191929 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:49.191475 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-85d467fcb7-bn4gl" Jul 23 00:17:49.209612 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:49.209541 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-85d467fcb7-bn4gl" podStartSLOduration=2.081023448 podStartE2EDuration="3.209517993s" totalImagesPullingTime="1.128494545s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:17:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:17:49.207592335 +0000 UTC m=+583.003099194" watchObservedRunningTime="2026-07-23 00:17:49.209517993 +0000 UTC m=+583.005024851" Jul 23 00:17:54.649117 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:54.649072 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678bfjwm"] Jul 23 00:17:54.653060 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:54.653023 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678bfjwm" Jul 23 00:17:54.656192 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:54.656161 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 23 00:17:54.656333 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:54.656250 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-pkk69\"" Jul 23 00:17:54.664013 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:54.663982 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678bfjwm"] Jul 23 00:17:54.680733 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:54.680693 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5b4d97f-58c7-4032-85f5-8c31e65c8947-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678bfjwm\" (UID: \"a5b4d97f-58c7-4032-85f5-8c31e65c8947\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678bfjwm" Jul 23 00:17:54.680972 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:54.680739 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5b4d97f-58c7-4032-85f5-8c31e65c8947-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678bfjwm\" (UID: \"a5b4d97f-58c7-4032-85f5-8c31e65c8947\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678bfjwm" Jul 23 00:17:54.680972 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:54.680802 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5b4d97f-58c7-4032-85f5-8c31e65c8947-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678bfjwm\" (UID: \"a5b4d97f-58c7-4032-85f5-8c31e65c8947\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678bfjwm" Jul 23 00:17:54.680972 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:54.680829 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vnfn\" (UniqueName: \"kubernetes.io/projected/a5b4d97f-58c7-4032-85f5-8c31e65c8947-kube-api-access-6vnfn\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678bfjwm\" (UID: \"a5b4d97f-58c7-4032-85f5-8c31e65c8947\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678bfjwm" Jul 23 00:17:54.680972 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:54.680863 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5b4d97f-58c7-4032-85f5-8c31e65c8947-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678bfjwm\" (UID: \"a5b4d97f-58c7-4032-85f5-8c31e65c8947\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678bfjwm" Jul 23 00:17:54.680972 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:54.680899 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5b4d97f-58c7-4032-85f5-8c31e65c8947-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678bfjwm\" (UID: \"a5b4d97f-58c7-4032-85f5-8c31e65c8947\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678bfjwm" Jul 23 00:17:54.681167 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:54.681004 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5b4d97f-58c7-4032-85f5-8c31e65c8947-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678bfjwm\" (UID: \"a5b4d97f-58c7-4032-85f5-8c31e65c8947\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678bfjwm" Jul 23 00:17:54.681167 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:54.681045 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5b4d97f-58c7-4032-85f5-8c31e65c8947-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678bfjwm\" (UID: \"a5b4d97f-58c7-4032-85f5-8c31e65c8947\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678bfjwm" Jul 23 00:17:54.681167 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:54.681066 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5b4d97f-58c7-4032-85f5-8c31e65c8947-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678bfjwm\" (UID: \"a5b4d97f-58c7-4032-85f5-8c31e65c8947\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678bfjwm" Jul 23 00:17:54.749139 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:54.749101 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-t62mx"] Jul 23 00:17:54.751584 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:54.751562 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-t62mx" Jul 23 00:17:54.754869 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:54.754832 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-5hjrl\"" Jul 23 00:17:54.755038 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:54.755004 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 23 00:17:54.755038 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:54.755016 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 23 00:17:54.755164 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:54.755020 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 23 00:17:54.764614 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:54.764583 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-t62mx"] Jul 23 00:17:54.782248 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:54.782213 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/557397d5-d277-4ee8-a5bd-bbc4fec09acb-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-t62mx\" (UID: \"557397d5-d277-4ee8-a5bd-bbc4fec09acb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-t62mx" Jul 23 00:17:54.782486 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:54.782276 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5b4d97f-58c7-4032-85f5-8c31e65c8947-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678bfjwm\" (UID: \"a5b4d97f-58c7-4032-85f5-8c31e65c8947\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678bfjwm" Jul 23 00:17:54.782486 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:54.782310 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5b4d97f-58c7-4032-85f5-8c31e65c8947-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678bfjwm\" (UID: \"a5b4d97f-58c7-4032-85f5-8c31e65c8947\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678bfjwm" Jul 23 00:17:54.782486 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:54.782340 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5b4d97f-58c7-4032-85f5-8c31e65c8947-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678bfjwm\" (UID: \"a5b4d97f-58c7-4032-85f5-8c31e65c8947\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678bfjwm" Jul 23 00:17:54.782486 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:54.782373 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wwp6\" (UniqueName: \"kubernetes.io/projected/557397d5-d277-4ee8-a5bd-bbc4fec09acb-kube-api-access-2wwp6\") pod \"istiod-openshift-gateway-94bb8fbfd-t62mx\" (UID: \"557397d5-d277-4ee8-a5bd-bbc4fec09acb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-t62mx" Jul 23 00:17:54.782486 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:54.782419 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/557397d5-d277-4ee8-a5bd-bbc4fec09acb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-t62mx\" (UID: \"557397d5-d277-4ee8-a5bd-bbc4fec09acb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-t62mx" Jul 23 00:17:54.782797 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:54.782500 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5b4d97f-58c7-4032-85f5-8c31e65c8947-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678bfjwm\" (UID: \"a5b4d97f-58c7-4032-85f5-8c31e65c8947\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678bfjwm" Jul 23 00:17:54.782797 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:54.782541 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5b4d97f-58c7-4032-85f5-8c31e65c8947-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678bfjwm\" (UID: \"a5b4d97f-58c7-4032-85f5-8c31e65c8947\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678bfjwm" Jul 23 00:17:54.782797 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:54.782570 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5b4d97f-58c7-4032-85f5-8c31e65c8947-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678bfjwm\" (UID: \"a5b4d97f-58c7-4032-85f5-8c31e65c8947\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678bfjwm" Jul 23 00:17:54.782797 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:54.782590 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/557397d5-d277-4ee8-a5bd-bbc4fec09acb-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-t62mx\" (UID: \"557397d5-d277-4ee8-a5bd-bbc4fec09acb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-t62mx" Jul 23 00:17:54.782797 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:54.782621 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/557397d5-d277-4ee8-a5bd-bbc4fec09acb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-t62mx\" (UID: \"557397d5-d277-4ee8-a5bd-bbc4fec09acb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-t62mx" Jul 23 00:17:54.782797 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:54.782643 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vnfn\" (UniqueName: \"kubernetes.io/projected/a5b4d97f-58c7-4032-85f5-8c31e65c8947-kube-api-access-6vnfn\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678bfjwm\" (UID: \"a5b4d97f-58c7-4032-85f5-8c31e65c8947\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678bfjwm" Jul 23 00:17:54.782797 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:54.782664 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/557397d5-d277-4ee8-a5bd-bbc4fec09acb-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-t62mx\" (UID: \"557397d5-d277-4ee8-a5bd-bbc4fec09acb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-t62mx" Jul 23 00:17:54.782797 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:54.782685 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/557397d5-d277-4ee8-a5bd-bbc4fec09acb-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-t62mx\" (UID: \"557397d5-d277-4ee8-a5bd-bbc4fec09acb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-t62mx" Jul 23 00:17:54.782797 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:54.782711 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5b4d97f-58c7-4032-85f5-8c31e65c8947-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678bfjwm\" (UID: \"a5b4d97f-58c7-4032-85f5-8c31e65c8947\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678bfjwm" Jul 23 00:17:54.782797 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:54.782729 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5b4d97f-58c7-4032-85f5-8c31e65c8947-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678bfjwm\" (UID: \"a5b4d97f-58c7-4032-85f5-8c31e65c8947\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678bfjwm" Jul 23 00:17:54.783352 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:54.782874 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5b4d97f-58c7-4032-85f5-8c31e65c8947-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678bfjwm\" (UID: \"a5b4d97f-58c7-4032-85f5-8c31e65c8947\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678bfjwm" Jul 23 00:17:54.783629 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:54.783597 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5b4d97f-58c7-4032-85f5-8c31e65c8947-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678bfjwm\" (UID: \"a5b4d97f-58c7-4032-85f5-8c31e65c8947\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678bfjwm" Jul 23 00:17:54.783929 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:54.783906 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5b4d97f-58c7-4032-85f5-8c31e65c8947-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678bfjwm\" (UID: \"a5b4d97f-58c7-4032-85f5-8c31e65c8947\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678bfjwm" Jul 23 00:17:54.784208 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:54.784188 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5b4d97f-58c7-4032-85f5-8c31e65c8947-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678bfjwm\" (UID: \"a5b4d97f-58c7-4032-85f5-8c31e65c8947\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678bfjwm" Jul 23 00:17:54.784314 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:54.784227 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5b4d97f-58c7-4032-85f5-8c31e65c8947-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678bfjwm\" (UID: \"a5b4d97f-58c7-4032-85f5-8c31e65c8947\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678bfjwm" Jul 23 00:17:54.785609 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:54.785548 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5b4d97f-58c7-4032-85f5-8c31e65c8947-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678bfjwm\" (UID: \"a5b4d97f-58c7-4032-85f5-8c31e65c8947\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678bfjwm" Jul 23 00:17:54.786381 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:54.786262 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5b4d97f-58c7-4032-85f5-8c31e65c8947-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678bfjwm\" (UID: \"a5b4d97f-58c7-4032-85f5-8c31e65c8947\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678bfjwm" Jul 23 00:17:54.793734 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:54.793689 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5b4d97f-58c7-4032-85f5-8c31e65c8947-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678bfjwm\" (UID: \"a5b4d97f-58c7-4032-85f5-8c31e65c8947\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678bfjwm" Jul 23 00:17:54.794001 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:54.793978 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vnfn\" (UniqueName: \"kubernetes.io/projected/a5b4d97f-58c7-4032-85f5-8c31e65c8947-kube-api-access-6vnfn\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678bfjwm\" (UID: \"a5b4d97f-58c7-4032-85f5-8c31e65c8947\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678bfjwm" Jul 23 00:17:54.883750 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:54.883706 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/557397d5-d277-4ee8-a5bd-bbc4fec09acb-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-t62mx\" (UID: \"557397d5-d277-4ee8-a5bd-bbc4fec09acb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-t62mx" Jul 23 00:17:54.883981 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:54.883784 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/557397d5-d277-4ee8-a5bd-bbc4fec09acb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-t62mx\" (UID: \"557397d5-d277-4ee8-a5bd-bbc4fec09acb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-t62mx" Jul 23 00:17:54.883981 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:54.883819 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/557397d5-d277-4ee8-a5bd-bbc4fec09acb-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-t62mx\" (UID: \"557397d5-d277-4ee8-a5bd-bbc4fec09acb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-t62mx" Jul 23 00:17:54.883981 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:54.883839 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/557397d5-d277-4ee8-a5bd-bbc4fec09acb-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-t62mx\" (UID: \"557397d5-d277-4ee8-a5bd-bbc4fec09acb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-t62mx" Jul 23 00:17:54.883981 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:54.883900 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/557397d5-d277-4ee8-a5bd-bbc4fec09acb-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-t62mx\" (UID: \"557397d5-d277-4ee8-a5bd-bbc4fec09acb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-t62mx" Jul 23 00:17:54.883981 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:54.883947 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wwp6\" (UniqueName: \"kubernetes.io/projected/557397d5-d277-4ee8-a5bd-bbc4fec09acb-kube-api-access-2wwp6\") pod \"istiod-openshift-gateway-94bb8fbfd-t62mx\" (UID: \"557397d5-d277-4ee8-a5bd-bbc4fec09acb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-t62mx" Jul 23 00:17:54.883981 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:54.883978 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/557397d5-d277-4ee8-a5bd-bbc4fec09acb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-t62mx\" (UID: \"557397d5-d277-4ee8-a5bd-bbc4fec09acb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-t62mx" Jul 23 00:17:54.884720 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:54.884658 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/557397d5-d277-4ee8-a5bd-bbc4fec09acb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-t62mx\" (UID: \"557397d5-d277-4ee8-a5bd-bbc4fec09acb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-t62mx" Jul 23 00:17:54.886496 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:54.886451 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/557397d5-d277-4ee8-a5bd-bbc4fec09acb-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-t62mx\" (UID: \"557397d5-d277-4ee8-a5bd-bbc4fec09acb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-t62mx" Jul 23 00:17:54.886713 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:54.886693 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/557397d5-d277-4ee8-a5bd-bbc4fec09acb-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-t62mx\" (UID: \"557397d5-d277-4ee8-a5bd-bbc4fec09acb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-t62mx" Jul 23 00:17:54.886855 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:54.886835 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/557397d5-d277-4ee8-a5bd-bbc4fec09acb-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-t62mx\" (UID: \"557397d5-d277-4ee8-a5bd-bbc4fec09acb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-t62mx" Jul 23 00:17:54.887412 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:54.887391 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/557397d5-d277-4ee8-a5bd-bbc4fec09acb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-t62mx\" (UID: \"557397d5-d277-4ee8-a5bd-bbc4fec09acb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-t62mx" Jul 23 00:17:54.904702 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:54.904621 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/557397d5-d277-4ee8-a5bd-bbc4fec09acb-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-t62mx\" (UID: \"557397d5-d277-4ee8-a5bd-bbc4fec09acb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-t62mx" Jul 23 00:17:54.904702 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:54.904668 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wwp6\" (UniqueName: \"kubernetes.io/projected/557397d5-d277-4ee8-a5bd-bbc4fec09acb-kube-api-access-2wwp6\") pod \"istiod-openshift-gateway-94bb8fbfd-t62mx\" (UID: \"557397d5-d277-4ee8-a5bd-bbc4fec09acb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-t62mx" Jul 23 00:17:54.965170 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:54.965128 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678bfjwm" Jul 23 00:17:55.062353 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:55.062315 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-t62mx" Jul 23 00:17:55.133379 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:55.132383 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678bfjwm"] Jul 23 00:17:55.135803 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:17:55.135749 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5b4d97f_58c7_4032_85f5_8c31e65c8947.slice/crio-c8b23343bab81f9aa5880117cd66eb54d32291b24a3cf8bf1e638c5f4c859c78 WatchSource:0}: Error finding container c8b23343bab81f9aa5880117cd66eb54d32291b24a3cf8bf1e638c5f4c859c78: Status 404 returned error can't find the container with id c8b23343bab81f9aa5880117cd66eb54d32291b24a3cf8bf1e638c5f4c859c78 Jul 23 00:17:55.225125 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:55.225085 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678bfjwm" event={"ID":"a5b4d97f-58c7-4032-85f5-8c31e65c8947","Type":"ContainerStarted","Data":"c8b23343bab81f9aa5880117cd66eb54d32291b24a3cf8bf1e638c5f4c859c78"} Jul 23 00:17:55.237749 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:55.233374 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-t62mx"] Jul 23 00:17:55.238717 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:17:55.238061 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod557397d5_d277_4ee8_a5bd_bbc4fec09acb.slice/crio-771a19b7c81d8de9987be0c52ced8d7fc46e3d260a7b730fdd1020f676962164 WatchSource:0}: Error finding container 771a19b7c81d8de9987be0c52ced8d7fc46e3d260a7b730fdd1020f676962164: Status 404 returned error can't find the container with id 771a19b7c81d8de9987be0c52ced8d7fc46e3d260a7b730fdd1020f676962164 Jul 23 00:17:56.233587 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:56.233537 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-t62mx" event={"ID":"557397d5-d277-4ee8-a5bd-bbc4fec09acb","Type":"ContainerStarted","Data":"771a19b7c81d8de9987be0c52ced8d7fc46e3d260a7b730fdd1020f676962164"} Jul 23 00:17:58.744038 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:58.744002 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:17:58.744361 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:58.744073 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:17:58.749268 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:58.749232 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:17:58.749417 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:58.749305 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:17:58.749417 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:58.749335 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:17:59.255142 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:59.255095 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-t62mx" event={"ID":"557397d5-d277-4ee8-a5bd-bbc4fec09acb","Type":"ContainerStarted","Data":"516229b58316fe3103957a4b2ed54a982226ead7d8e9c284e2d0d6c24bec5027"} Jul 23 00:17:59.255362 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:59.255170 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-t62mx" Jul 23 00:17:59.257457 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:59.257405 2562 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-t62mx container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 23 00:17:59.257610 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:59.257465 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-t62mx" podUID="557397d5-d277-4ee8-a5bd-bbc4fec09acb" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 23 00:17:59.257610 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:59.257484 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678bfjwm" event={"ID":"a5b4d97f-58c7-4032-85f5-8c31e65c8947","Type":"ContainerStarted","Data":"291c496d93598679059a9bda7cdd3b65b421d06e410b3df608de25d27949b25d"} Jul 23 00:17:59.295004 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:59.294942 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-t62mx" podStartSLOduration=1.7926136879999999 podStartE2EDuration="5.294922112s" totalImagesPullingTime="3.502308424s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:17:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:17:59.294902141 +0000 UTC m=+593.090409012" watchObservedRunningTime="2026-07-23 00:17:59.294922112 +0000 UTC m=+593.090428966" Jul 23 00:17:59.320836 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:59.320748 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678bfjwm" podStartSLOduration=1.710368635 podStartE2EDuration="5.320729233s" totalImagesPullingTime="3.610360598s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:17:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:17:59.320673011 +0000 UTC m=+593.116179868" watchObservedRunningTime="2026-07-23 00:17:59.320729233 +0000 UTC m=+593.116236091" Jul 23 00:17:59.965506 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:59.965465 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678bfjwm" Jul 23 00:17:59.971312 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:17:59.971280 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678bfjwm" Jul 23 00:18:00.199941 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:18:00.199902 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-85d467fcb7-bn4gl" Jul 23 00:18:00.265033 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:18:00.264815 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678bfjwm" Jul 23 00:18:00.265273 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:18:00.265255 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-t62mx" Jul 23 00:18:00.266051 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:18:00.266016 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678bfjwm" Jul 23 00:19:03.497335 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:03.497248 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hw9f7"] Jul 23 00:19:03.502770 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:03.502728 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hw9f7" Jul 23 00:19:03.507024 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:03.506946 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-gfxkf\"" Jul 23 00:19:03.507194 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:03.507073 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 23 00:19:03.507265 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:03.507236 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 23 00:19:03.509655 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:03.509625 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hw9f7"] Jul 23 00:19:03.530367 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:03.530332 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc5b0c3a-51dc-4e24-a6de-140c0ecb8f24-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hw9f7\" (UID: \"cc5b0c3a-51dc-4e24-a6de-140c0ecb8f24\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hw9f7" Jul 23 00:19:03.530367 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:03.530374 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc5b0c3a-51dc-4e24-a6de-140c0ecb8f24-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hw9f7\" (UID: \"cc5b0c3a-51dc-4e24-a6de-140c0ecb8f24\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hw9f7" Jul 23 00:19:03.530584 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:03.530414 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnhhs\" (UniqueName: \"kubernetes.io/projected/cc5b0c3a-51dc-4e24-a6de-140c0ecb8f24-kube-api-access-bnhhs\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hw9f7\" (UID: \"cc5b0c3a-51dc-4e24-a6de-140c0ecb8f24\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hw9f7" Jul 23 00:19:03.631480 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:03.631442 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc5b0c3a-51dc-4e24-a6de-140c0ecb8f24-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hw9f7\" (UID: \"cc5b0c3a-51dc-4e24-a6de-140c0ecb8f24\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hw9f7" Jul 23 00:19:03.631480 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:03.631486 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc5b0c3a-51dc-4e24-a6de-140c0ecb8f24-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hw9f7\" (UID: \"cc5b0c3a-51dc-4e24-a6de-140c0ecb8f24\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hw9f7" Jul 23 00:19:03.631724 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:03.631514 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnhhs\" (UniqueName: \"kubernetes.io/projected/cc5b0c3a-51dc-4e24-a6de-140c0ecb8f24-kube-api-access-bnhhs\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hw9f7\" (UID: \"cc5b0c3a-51dc-4e24-a6de-140c0ecb8f24\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hw9f7" Jul 23 00:19:03.631922 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:03.631902 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc5b0c3a-51dc-4e24-a6de-140c0ecb8f24-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hw9f7\" (UID: \"cc5b0c3a-51dc-4e24-a6de-140c0ecb8f24\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hw9f7" Jul 23 00:19:03.631968 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:03.631920 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc5b0c3a-51dc-4e24-a6de-140c0ecb8f24-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hw9f7\" (UID: \"cc5b0c3a-51dc-4e24-a6de-140c0ecb8f24\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hw9f7" Jul 23 00:19:03.640440 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:03.640413 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnhhs\" (UniqueName: \"kubernetes.io/projected/cc5b0c3a-51dc-4e24-a6de-140c0ecb8f24-kube-api-access-bnhhs\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hw9f7\" (UID: \"cc5b0c3a-51dc-4e24-a6de-140c0ecb8f24\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hw9f7" Jul 23 00:19:03.815473 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:03.815430 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hw9f7" Jul 23 00:19:03.959186 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:03.959157 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hw9f7"] Jul 23 00:19:03.960912 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:19:03.960875 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc5b0c3a_51dc_4e24_a6de_140c0ecb8f24.slice/crio-265888c6e8065ef607ff7841401b569f13d69cc6634c7a3826711abd8f79ccf7 WatchSource:0}: Error finding container 265888c6e8065ef607ff7841401b569f13d69cc6634c7a3826711abd8f79ccf7: Status 404 returned error can't find the container with id 265888c6e8065ef607ff7841401b569f13d69cc6634c7a3826711abd8f79ccf7 Jul 23 00:19:04.296474 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:04.296434 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nn4k6"] Jul 23 00:19:04.299460 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:04.299435 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nn4k6" Jul 23 00:19:04.308500 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:04.308474 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nn4k6"] Jul 23 00:19:04.337409 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:04.337308 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e929649-eea1-4da3-ab01-a77d5c4a0d34-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nn4k6\" (UID: \"2e929649-eea1-4da3-ab01-a77d5c4a0d34\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nn4k6" Jul 23 00:19:04.337409 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:04.337356 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlcnv\" (UniqueName: \"kubernetes.io/projected/2e929649-eea1-4da3-ab01-a77d5c4a0d34-kube-api-access-vlcnv\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nn4k6\" (UID: \"2e929649-eea1-4da3-ab01-a77d5c4a0d34\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nn4k6" Jul 23 00:19:04.337604 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:04.337508 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e929649-eea1-4da3-ab01-a77d5c4a0d34-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nn4k6\" (UID: \"2e929649-eea1-4da3-ab01-a77d5c4a0d34\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nn4k6" Jul 23 00:19:04.439024 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:04.438985 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e929649-eea1-4da3-ab01-a77d5c4a0d34-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nn4k6\" (UID: \"2e929649-eea1-4da3-ab01-a77d5c4a0d34\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nn4k6" Jul 23 00:19:04.439245 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:04.439054 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e929649-eea1-4da3-ab01-a77d5c4a0d34-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nn4k6\" (UID: \"2e929649-eea1-4da3-ab01-a77d5c4a0d34\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nn4k6" Jul 23 00:19:04.439245 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:04.439076 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlcnv\" (UniqueName: \"kubernetes.io/projected/2e929649-eea1-4da3-ab01-a77d5c4a0d34-kube-api-access-vlcnv\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nn4k6\" (UID: \"2e929649-eea1-4da3-ab01-a77d5c4a0d34\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nn4k6" Jul 23 00:19:04.439515 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:04.439491 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e929649-eea1-4da3-ab01-a77d5c4a0d34-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nn4k6\" (UID: \"2e929649-eea1-4da3-ab01-a77d5c4a0d34\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nn4k6" Jul 23 00:19:04.439582 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:04.439492 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e929649-eea1-4da3-ab01-a77d5c4a0d34-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nn4k6\" (UID: \"2e929649-eea1-4da3-ab01-a77d5c4a0d34\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nn4k6" Jul 23 00:19:04.448125 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:04.448094 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlcnv\" (UniqueName: \"kubernetes.io/projected/2e929649-eea1-4da3-ab01-a77d5c4a0d34-kube-api-access-vlcnv\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nn4k6\" (UID: \"2e929649-eea1-4da3-ab01-a77d5c4a0d34\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nn4k6" Jul 23 00:19:04.585265 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:04.585227 2562 generic.go:356] "Generic (PLEG): container finished" podID="cc5b0c3a-51dc-4e24-a6de-140c0ecb8f24" containerID="51c81c27d43af8d4b97e5696eccc81c8b94742018f6a7faf7f3134d71743f479" exitCode=0 Jul 23 00:19:04.585734 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:04.585287 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hw9f7" event={"ID":"cc5b0c3a-51dc-4e24-a6de-140c0ecb8f24","Type":"ContainerDied","Data":"51c81c27d43af8d4b97e5696eccc81c8b94742018f6a7faf7f3134d71743f479"} Jul 23 00:19:04.585734 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:04.585330 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hw9f7" event={"ID":"cc5b0c3a-51dc-4e24-a6de-140c0ecb8f24","Type":"ContainerStarted","Data":"265888c6e8065ef607ff7841401b569f13d69cc6634c7a3826711abd8f79ccf7"} Jul 23 00:19:04.610582 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:04.610535 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nn4k6" Jul 23 00:19:04.752061 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:04.752033 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nn4k6"] Jul 23 00:19:04.754197 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:19:04.754163 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e929649_eea1_4da3_ab01_a77d5c4a0d34.slice/crio-238effd07fdbeb73756aa0057a00080fc7375a5f44709a3b43eb211692fdaad1 WatchSource:0}: Error finding container 238effd07fdbeb73756aa0057a00080fc7375a5f44709a3b43eb211692fdaad1: Status 404 returned error can't find the container with id 238effd07fdbeb73756aa0057a00080fc7375a5f44709a3b43eb211692fdaad1 Jul 23 00:19:04.894341 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:04.894299 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pxctb"] Jul 23 00:19:04.897511 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:04.897489 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pxctb" Jul 23 00:19:04.905396 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:04.905364 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pxctb"] Jul 23 00:19:04.943778 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:04.943732 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5495p\" (UniqueName: \"kubernetes.io/projected/a19a892d-c6f3-451e-b60b-824e30993eae-kube-api-access-5495p\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pxctb\" (UID: \"a19a892d-c6f3-451e-b60b-824e30993eae\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pxctb" Jul 23 00:19:04.943921 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:04.943827 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a19a892d-c6f3-451e-b60b-824e30993eae-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pxctb\" (UID: \"a19a892d-c6f3-451e-b60b-824e30993eae\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pxctb" Jul 23 00:19:04.943921 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:04.943890 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a19a892d-c6f3-451e-b60b-824e30993eae-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pxctb\" (UID: \"a19a892d-c6f3-451e-b60b-824e30993eae\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pxctb" Jul 23 00:19:05.044722 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:05.044673 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a19a892d-c6f3-451e-b60b-824e30993eae-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pxctb\" (UID: \"a19a892d-c6f3-451e-b60b-824e30993eae\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pxctb" Jul 23 00:19:05.044722 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:05.044723 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a19a892d-c6f3-451e-b60b-824e30993eae-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pxctb\" (UID: \"a19a892d-c6f3-451e-b60b-824e30993eae\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pxctb" Jul 23 00:19:05.045017 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:05.044988 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5495p\" (UniqueName: \"kubernetes.io/projected/a19a892d-c6f3-451e-b60b-824e30993eae-kube-api-access-5495p\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pxctb\" (UID: \"a19a892d-c6f3-451e-b60b-824e30993eae\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pxctb" Jul 23 00:19:05.045170 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:05.045150 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a19a892d-c6f3-451e-b60b-824e30993eae-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pxctb\" (UID: \"a19a892d-c6f3-451e-b60b-824e30993eae\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pxctb" Jul 23 00:19:05.045247 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:05.045203 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a19a892d-c6f3-451e-b60b-824e30993eae-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pxctb\" (UID: \"a19a892d-c6f3-451e-b60b-824e30993eae\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pxctb" Jul 23 00:19:05.053393 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:05.053362 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5495p\" (UniqueName: \"kubernetes.io/projected/a19a892d-c6f3-451e-b60b-824e30993eae-kube-api-access-5495p\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pxctb\" (UID: \"a19a892d-c6f3-451e-b60b-824e30993eae\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pxctb" Jul 23 00:19:05.242286 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:05.242200 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pxctb" Jul 23 00:19:05.295139 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:05.295101 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nm6kj"] Jul 23 00:19:05.299518 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:05.299496 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nm6kj" Jul 23 00:19:05.307301 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:05.307268 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nm6kj"] Jul 23 00:19:05.347525 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:05.347470 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3299b5b8-cd31-4b1f-9f76-dcbf5b1f5337-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nm6kj\" (UID: \"3299b5b8-cd31-4b1f-9f76-dcbf5b1f5337\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nm6kj" Jul 23 00:19:05.347525 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:05.347526 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j44p4\" (UniqueName: \"kubernetes.io/projected/3299b5b8-cd31-4b1f-9f76-dcbf5b1f5337-kube-api-access-j44p4\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nm6kj\" (UID: \"3299b5b8-cd31-4b1f-9f76-dcbf5b1f5337\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nm6kj" Jul 23 00:19:05.347791 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:05.347617 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3299b5b8-cd31-4b1f-9f76-dcbf5b1f5337-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nm6kj\" (UID: \"3299b5b8-cd31-4b1f-9f76-dcbf5b1f5337\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nm6kj" Jul 23 00:19:05.395647 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:05.395618 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pxctb"] Jul 23 00:19:05.448633 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:05.448596 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3299b5b8-cd31-4b1f-9f76-dcbf5b1f5337-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nm6kj\" (UID: \"3299b5b8-cd31-4b1f-9f76-dcbf5b1f5337\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nm6kj" Jul 23 00:19:05.448633 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:05.448637 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j44p4\" (UniqueName: \"kubernetes.io/projected/3299b5b8-cd31-4b1f-9f76-dcbf5b1f5337-kube-api-access-j44p4\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nm6kj\" (UID: \"3299b5b8-cd31-4b1f-9f76-dcbf5b1f5337\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nm6kj" Jul 23 00:19:05.448899 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:05.448678 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3299b5b8-cd31-4b1f-9f76-dcbf5b1f5337-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nm6kj\" (UID: \"3299b5b8-cd31-4b1f-9f76-dcbf5b1f5337\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nm6kj" Jul 23 00:19:05.449006 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:05.448986 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3299b5b8-cd31-4b1f-9f76-dcbf5b1f5337-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nm6kj\" (UID: \"3299b5b8-cd31-4b1f-9f76-dcbf5b1f5337\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nm6kj" Jul 23 00:19:05.449065 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:05.449019 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3299b5b8-cd31-4b1f-9f76-dcbf5b1f5337-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nm6kj\" (UID: \"3299b5b8-cd31-4b1f-9f76-dcbf5b1f5337\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nm6kj" Jul 23 00:19:05.458787 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:05.458725 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j44p4\" (UniqueName: \"kubernetes.io/projected/3299b5b8-cd31-4b1f-9f76-dcbf5b1f5337-kube-api-access-j44p4\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nm6kj\" (UID: \"3299b5b8-cd31-4b1f-9f76-dcbf5b1f5337\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nm6kj" Jul 23 00:19:05.477435 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:19:05.477398 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda19a892d_c6f3_451e_b60b_824e30993eae.slice/crio-91e18dd755f4304cb37314b26f0d00f73857b8cec1648e7d66d52da8ddec0612 WatchSource:0}: Error finding container 91e18dd755f4304cb37314b26f0d00f73857b8cec1648e7d66d52da8ddec0612: Status 404 returned error can't find the container with id 91e18dd755f4304cb37314b26f0d00f73857b8cec1648e7d66d52da8ddec0612 Jul 23 00:19:05.592400 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:05.592361 2562 generic.go:356] "Generic (PLEG): container finished" podID="2e929649-eea1-4da3-ab01-a77d5c4a0d34" containerID="9e98f0d74e9e1f324c98eb7674fae4d9a1252b2713594242a28cc17d9bd53c28" exitCode=0 Jul 23 00:19:05.592808 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:05.592440 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nn4k6" event={"ID":"2e929649-eea1-4da3-ab01-a77d5c4a0d34","Type":"ContainerDied","Data":"9e98f0d74e9e1f324c98eb7674fae4d9a1252b2713594242a28cc17d9bd53c28"} Jul 23 00:19:05.592808 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:05.592488 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nn4k6" event={"ID":"2e929649-eea1-4da3-ab01-a77d5c4a0d34","Type":"ContainerStarted","Data":"238effd07fdbeb73756aa0057a00080fc7375a5f44709a3b43eb211692fdaad1"} Jul 23 00:19:05.594213 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:05.594184 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pxctb" event={"ID":"a19a892d-c6f3-451e-b60b-824e30993eae","Type":"ContainerStarted","Data":"621576fec9455589f6263bed8f7658a64ebdc474be385f35829bd580be0be8b3"} Jul 23 00:19:05.594324 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:05.594233 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pxctb" event={"ID":"a19a892d-c6f3-451e-b60b-824e30993eae","Type":"ContainerStarted","Data":"91e18dd755f4304cb37314b26f0d00f73857b8cec1648e7d66d52da8ddec0612"} Jul 23 00:19:05.615023 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:05.614990 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nm6kj" Jul 23 00:19:05.763189 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:05.763155 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nm6kj"] Jul 23 00:19:05.765256 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:19:05.765208 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3299b5b8_cd31_4b1f_9f76_dcbf5b1f5337.slice/crio-55896c4d2f4716f0f36260e4fe441e90bbdfd14cacb5b52155d1cae7ddccbcaa WatchSource:0}: Error finding container 55896c4d2f4716f0f36260e4fe441e90bbdfd14cacb5b52155d1cae7ddccbcaa: Status 404 returned error can't find the container with id 55896c4d2f4716f0f36260e4fe441e90bbdfd14cacb5b52155d1cae7ddccbcaa Jul 23 00:19:06.608837 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:06.608780 2562 generic.go:356] "Generic (PLEG): container finished" podID="cc5b0c3a-51dc-4e24-a6de-140c0ecb8f24" containerID="bc4c246e5a719910d8b5b42326fc92b77faf961dfde1441872142943a6bf116b" exitCode=0 Jul 23 00:19:06.609286 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:06.608801 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hw9f7" event={"ID":"cc5b0c3a-51dc-4e24-a6de-140c0ecb8f24","Type":"ContainerDied","Data":"bc4c246e5a719910d8b5b42326fc92b77faf961dfde1441872142943a6bf116b"} Jul 23 00:19:06.611155 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:06.611124 2562 generic.go:356] "Generic (PLEG): container finished" podID="2e929649-eea1-4da3-ab01-a77d5c4a0d34" containerID="973c0c04c5bf855b8387849bbcbf3dd6d5c5847a08d011c0ca3dd461c1f86a97" exitCode=0 Jul 23 00:19:06.611264 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:06.611200 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nn4k6" event={"ID":"2e929649-eea1-4da3-ab01-a77d5c4a0d34","Type":"ContainerDied","Data":"973c0c04c5bf855b8387849bbcbf3dd6d5c5847a08d011c0ca3dd461c1f86a97"} Jul 23 00:19:06.612946 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:06.612919 2562 generic.go:356] "Generic (PLEG): container finished" podID="a19a892d-c6f3-451e-b60b-824e30993eae" containerID="621576fec9455589f6263bed8f7658a64ebdc474be385f35829bd580be0be8b3" exitCode=0 Jul 23 00:19:06.613054 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:06.612989 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pxctb" event={"ID":"a19a892d-c6f3-451e-b60b-824e30993eae","Type":"ContainerDied","Data":"621576fec9455589f6263bed8f7658a64ebdc474be385f35829bd580be0be8b3"} Jul 23 00:19:06.614697 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:06.614671 2562 generic.go:356] "Generic (PLEG): container finished" podID="3299b5b8-cd31-4b1f-9f76-dcbf5b1f5337" containerID="b8eab2df1240f8c20c858b1d8b6a9e5a533abb36b40956b399c2420601104bca" exitCode=0 Jul 23 00:19:06.614852 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:06.614727 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nm6kj" event={"ID":"3299b5b8-cd31-4b1f-9f76-dcbf5b1f5337","Type":"ContainerDied","Data":"b8eab2df1240f8c20c858b1d8b6a9e5a533abb36b40956b399c2420601104bca"} Jul 23 00:19:06.614852 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:06.614790 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nm6kj" event={"ID":"3299b5b8-cd31-4b1f-9f76-dcbf5b1f5337","Type":"ContainerStarted","Data":"55896c4d2f4716f0f36260e4fe441e90bbdfd14cacb5b52155d1cae7ddccbcaa"} Jul 23 00:19:07.623109 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:07.623063 2562 generic.go:356] "Generic (PLEG): container finished" podID="cc5b0c3a-51dc-4e24-a6de-140c0ecb8f24" containerID="c7a3b81ba6450bd146a7c8bdda3522fcd7b75fb1f5d4aea49d09b21ff73c369a" exitCode=0 Jul 23 00:19:07.623571 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:07.623166 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hw9f7" event={"ID":"cc5b0c3a-51dc-4e24-a6de-140c0ecb8f24","Type":"ContainerDied","Data":"c7a3b81ba6450bd146a7c8bdda3522fcd7b75fb1f5d4aea49d09b21ff73c369a"} Jul 23 00:19:07.625484 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:07.625458 2562 generic.go:356] "Generic (PLEG): container finished" podID="2e929649-eea1-4da3-ab01-a77d5c4a0d34" containerID="5342ffd281d84e6f5b4f8a9ee00f5e26f822f0fcd5fec4222ef56fed1be5f7e0" exitCode=0 Jul 23 00:19:07.625629 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:07.625551 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nn4k6" event={"ID":"2e929649-eea1-4da3-ab01-a77d5c4a0d34","Type":"ContainerDied","Data":"5342ffd281d84e6f5b4f8a9ee00f5e26f822f0fcd5fec4222ef56fed1be5f7e0"} Jul 23 00:19:08.575286 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:19:08.575242 2562 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc5b0c3a_51dc_4e24_a6de_140c0ecb8f24.slice/crio-c7a3b81ba6450bd146a7c8bdda3522fcd7b75fb1f5d4aea49d09b21ff73c369a.scope\": RecentStats: unable to find data in memory cache]" Jul 23 00:19:08.632610 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:08.632573 2562 generic.go:356] "Generic (PLEG): container finished" podID="a19a892d-c6f3-451e-b60b-824e30993eae" containerID="96c0678f3547861914279deb41f209769806372d347c240be3654c6ba2a87ccf" exitCode=0 Jul 23 00:19:08.633069 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:08.632660 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pxctb" event={"ID":"a19a892d-c6f3-451e-b60b-824e30993eae","Type":"ContainerDied","Data":"96c0678f3547861914279deb41f209769806372d347c240be3654c6ba2a87ccf"} Jul 23 00:19:08.634808 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:08.634779 2562 generic.go:356] "Generic (PLEG): container finished" podID="3299b5b8-cd31-4b1f-9f76-dcbf5b1f5337" containerID="14ce2a2d63e65ce147127bbb0b04196b4eef27844b0dde30b638bdd5527d1893" exitCode=0 Jul 23 00:19:08.634882 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:08.634860 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nm6kj" event={"ID":"3299b5b8-cd31-4b1f-9f76-dcbf5b1f5337","Type":"ContainerDied","Data":"14ce2a2d63e65ce147127bbb0b04196b4eef27844b0dde30b638bdd5527d1893"} Jul 23 00:19:08.654956 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:19:08.654414 2562 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc5b0c3a_51dc_4e24_a6de_140c0ecb8f24.slice/crio-c7a3b81ba6450bd146a7c8bdda3522fcd7b75fb1f5d4aea49d09b21ff73c369a.scope\": RecentStats: unable to find data in memory cache]" Jul 23 00:19:08.816598 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:08.816535 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hw9f7" Jul 23 00:19:08.843572 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:08.843460 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nn4k6" Jul 23 00:19:08.886559 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:08.886520 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc5b0c3a-51dc-4e24-a6de-140c0ecb8f24-util\") pod \"cc5b0c3a-51dc-4e24-a6de-140c0ecb8f24\" (UID: \"cc5b0c3a-51dc-4e24-a6de-140c0ecb8f24\") " Jul 23 00:19:08.886712 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:08.886604 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vlcnv\" (UniqueName: \"kubernetes.io/projected/2e929649-eea1-4da3-ab01-a77d5c4a0d34-kube-api-access-vlcnv\") pod \"2e929649-eea1-4da3-ab01-a77d5c4a0d34\" (UID: \"2e929649-eea1-4da3-ab01-a77d5c4a0d34\") " Jul 23 00:19:08.886712 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:08.886641 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e929649-eea1-4da3-ab01-a77d5c4a0d34-util\") pod \"2e929649-eea1-4da3-ab01-a77d5c4a0d34\" (UID: \"2e929649-eea1-4da3-ab01-a77d5c4a0d34\") " Jul 23 00:19:08.886712 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:08.886675 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc5b0c3a-51dc-4e24-a6de-140c0ecb8f24-bundle\") pod \"cc5b0c3a-51dc-4e24-a6de-140c0ecb8f24\" (UID: \"cc5b0c3a-51dc-4e24-a6de-140c0ecb8f24\") " Jul 23 00:19:08.886931 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:08.886721 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e929649-eea1-4da3-ab01-a77d5c4a0d34-bundle\") pod \"2e929649-eea1-4da3-ab01-a77d5c4a0d34\" (UID: \"2e929649-eea1-4da3-ab01-a77d5c4a0d34\") " Jul 23 00:19:08.886931 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:08.886772 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bnhhs\" (UniqueName: \"kubernetes.io/projected/cc5b0c3a-51dc-4e24-a6de-140c0ecb8f24-kube-api-access-bnhhs\") pod \"cc5b0c3a-51dc-4e24-a6de-140c0ecb8f24\" (UID: \"cc5b0c3a-51dc-4e24-a6de-140c0ecb8f24\") " Jul 23 00:19:08.887185 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:08.887151 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc5b0c3a-51dc-4e24-a6de-140c0ecb8f24-bundle" (OuterVolumeSpecName: "bundle") pod "cc5b0c3a-51dc-4e24-a6de-140c0ecb8f24" (UID: "cc5b0c3a-51dc-4e24-a6de-140c0ecb8f24"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:19:08.887481 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:08.887450 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e929649-eea1-4da3-ab01-a77d5c4a0d34-bundle" (OuterVolumeSpecName: "bundle") pod "2e929649-eea1-4da3-ab01-a77d5c4a0d34" (UID: "2e929649-eea1-4da3-ab01-a77d5c4a0d34"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:19:08.889058 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:08.889025 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc5b0c3a-51dc-4e24-a6de-140c0ecb8f24-kube-api-access-bnhhs" (OuterVolumeSpecName: "kube-api-access-bnhhs") pod "cc5b0c3a-51dc-4e24-a6de-140c0ecb8f24" (UID: "cc5b0c3a-51dc-4e24-a6de-140c0ecb8f24"). InnerVolumeSpecName "kube-api-access-bnhhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:19:08.889152 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:08.889118 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e929649-eea1-4da3-ab01-a77d5c4a0d34-kube-api-access-vlcnv" (OuterVolumeSpecName: "kube-api-access-vlcnv") pod "2e929649-eea1-4da3-ab01-a77d5c4a0d34" (UID: "2e929649-eea1-4da3-ab01-a77d5c4a0d34"). InnerVolumeSpecName "kube-api-access-vlcnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:19:08.892749 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:08.892706 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e929649-eea1-4da3-ab01-a77d5c4a0d34-util" (OuterVolumeSpecName: "util") pod "2e929649-eea1-4da3-ab01-a77d5c4a0d34" (UID: "2e929649-eea1-4da3-ab01-a77d5c4a0d34"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:19:08.893839 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:08.893812 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc5b0c3a-51dc-4e24-a6de-140c0ecb8f24-util" (OuterVolumeSpecName: "util") pod "cc5b0c3a-51dc-4e24-a6de-140c0ecb8f24" (UID: "cc5b0c3a-51dc-4e24-a6de-140c0ecb8f24"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:19:08.987708 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:08.987668 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlcnv\" (UniqueName: \"kubernetes.io/projected/2e929649-eea1-4da3-ab01-a77d5c4a0d34-kube-api-access-vlcnv\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:19:08.987708 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:08.987703 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e929649-eea1-4da3-ab01-a77d5c4a0d34-util\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:19:08.987708 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:08.987717 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc5b0c3a-51dc-4e24-a6de-140c0ecb8f24-bundle\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:19:08.987979 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:08.987728 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e929649-eea1-4da3-ab01-a77d5c4a0d34-bundle\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:19:08.987979 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:08.987736 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnhhs\" (UniqueName: \"kubernetes.io/projected/cc5b0c3a-51dc-4e24-a6de-140c0ecb8f24-kube-api-access-bnhhs\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:19:08.987979 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:08.987745 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc5b0c3a-51dc-4e24-a6de-140c0ecb8f24-util\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:19:09.642547 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:09.642509 2562 generic.go:356] "Generic (PLEG): container finished" podID="a19a892d-c6f3-451e-b60b-824e30993eae" containerID="0784714e91e024f838b7ac49e6ea51dd691a003f04c0cce636ff0bb3e8c6654b" exitCode=0 Jul 23 00:19:09.642996 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:09.642582 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pxctb" event={"ID":"a19a892d-c6f3-451e-b60b-824e30993eae","Type":"ContainerDied","Data":"0784714e91e024f838b7ac49e6ea51dd691a003f04c0cce636ff0bb3e8c6654b"} Jul 23 00:19:09.644557 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:09.644530 2562 generic.go:356] "Generic (PLEG): container finished" podID="3299b5b8-cd31-4b1f-9f76-dcbf5b1f5337" containerID="01564e90c48caccc13f5e5dcb4491e82769348db715afc7538ad8c77841efe95" exitCode=0 Jul 23 00:19:09.644660 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:09.644572 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nm6kj" event={"ID":"3299b5b8-cd31-4b1f-9f76-dcbf5b1f5337","Type":"ContainerDied","Data":"01564e90c48caccc13f5e5dcb4491e82769348db715afc7538ad8c77841efe95"} Jul 23 00:19:09.646356 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:09.646333 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hw9f7" event={"ID":"cc5b0c3a-51dc-4e24-a6de-140c0ecb8f24","Type":"ContainerDied","Data":"265888c6e8065ef607ff7841401b569f13d69cc6634c7a3826711abd8f79ccf7"} Jul 23 00:19:09.646450 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:09.646352 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hw9f7" Jul 23 00:19:09.646450 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:09.646371 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="265888c6e8065ef607ff7841401b569f13d69cc6634c7a3826711abd8f79ccf7" Jul 23 00:19:09.648332 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:09.648308 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nn4k6" Jul 23 00:19:09.648455 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:09.648307 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nn4k6" event={"ID":"2e929649-eea1-4da3-ab01-a77d5c4a0d34","Type":"ContainerDied","Data":"238effd07fdbeb73756aa0057a00080fc7375a5f44709a3b43eb211692fdaad1"} Jul 23 00:19:09.648455 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:09.648449 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="238effd07fdbeb73756aa0057a00080fc7375a5f44709a3b43eb211692fdaad1" Jul 23 00:19:10.814474 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:10.814446 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nm6kj" Jul 23 00:19:10.816866 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:10.816843 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pxctb" Jul 23 00:19:10.906157 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:10.906065 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a19a892d-c6f3-451e-b60b-824e30993eae-util\") pod \"a19a892d-c6f3-451e-b60b-824e30993eae\" (UID: \"a19a892d-c6f3-451e-b60b-824e30993eae\") " Jul 23 00:19:10.906331 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:10.906172 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5495p\" (UniqueName: \"kubernetes.io/projected/a19a892d-c6f3-451e-b60b-824e30993eae-kube-api-access-5495p\") pod \"a19a892d-c6f3-451e-b60b-824e30993eae\" (UID: \"a19a892d-c6f3-451e-b60b-824e30993eae\") " Jul 23 00:19:10.906331 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:10.906208 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a19a892d-c6f3-451e-b60b-824e30993eae-bundle\") pod \"a19a892d-c6f3-451e-b60b-824e30993eae\" (UID: \"a19a892d-c6f3-451e-b60b-824e30993eae\") " Jul 23 00:19:10.906331 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:10.906228 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3299b5b8-cd31-4b1f-9f76-dcbf5b1f5337-util\") pod \"3299b5b8-cd31-4b1f-9f76-dcbf5b1f5337\" (UID: \"3299b5b8-cd31-4b1f-9f76-dcbf5b1f5337\") " Jul 23 00:19:10.906331 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:10.906266 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3299b5b8-cd31-4b1f-9f76-dcbf5b1f5337-bundle\") pod \"3299b5b8-cd31-4b1f-9f76-dcbf5b1f5337\" (UID: \"3299b5b8-cd31-4b1f-9f76-dcbf5b1f5337\") " Jul 23 00:19:10.906331 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:10.906300 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j44p4\" (UniqueName: \"kubernetes.io/projected/3299b5b8-cd31-4b1f-9f76-dcbf5b1f5337-kube-api-access-j44p4\") pod \"3299b5b8-cd31-4b1f-9f76-dcbf5b1f5337\" (UID: \"3299b5b8-cd31-4b1f-9f76-dcbf5b1f5337\") " Jul 23 00:19:10.906830 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:10.906799 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a19a892d-c6f3-451e-b60b-824e30993eae-bundle" (OuterVolumeSpecName: "bundle") pod "a19a892d-c6f3-451e-b60b-824e30993eae" (UID: "a19a892d-c6f3-451e-b60b-824e30993eae"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:19:10.906943 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:10.906920 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3299b5b8-cd31-4b1f-9f76-dcbf5b1f5337-bundle" (OuterVolumeSpecName: "bundle") pod "3299b5b8-cd31-4b1f-9f76-dcbf5b1f5337" (UID: "3299b5b8-cd31-4b1f-9f76-dcbf5b1f5337"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:19:10.908461 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:10.908429 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a19a892d-c6f3-451e-b60b-824e30993eae-kube-api-access-5495p" (OuterVolumeSpecName: "kube-api-access-5495p") pod "a19a892d-c6f3-451e-b60b-824e30993eae" (UID: "a19a892d-c6f3-451e-b60b-824e30993eae"). InnerVolumeSpecName "kube-api-access-5495p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:19:10.908461 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:10.908447 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3299b5b8-cd31-4b1f-9f76-dcbf5b1f5337-kube-api-access-j44p4" (OuterVolumeSpecName: "kube-api-access-j44p4") pod "3299b5b8-cd31-4b1f-9f76-dcbf5b1f5337" (UID: "3299b5b8-cd31-4b1f-9f76-dcbf5b1f5337"). InnerVolumeSpecName "kube-api-access-j44p4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:19:10.915003 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:10.914946 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a19a892d-c6f3-451e-b60b-824e30993eae-util" (OuterVolumeSpecName: "util") pod "a19a892d-c6f3-451e-b60b-824e30993eae" (UID: "a19a892d-c6f3-451e-b60b-824e30993eae"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:19:10.915178 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:10.915152 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3299b5b8-cd31-4b1f-9f76-dcbf5b1f5337-util" (OuterVolumeSpecName: "util") pod "3299b5b8-cd31-4b1f-9f76-dcbf5b1f5337" (UID: "3299b5b8-cd31-4b1f-9f76-dcbf5b1f5337"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:19:11.007270 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:11.007234 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a19a892d-c6f3-451e-b60b-824e30993eae-util\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:19:11.007270 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:11.007265 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5495p\" (UniqueName: \"kubernetes.io/projected/a19a892d-c6f3-451e-b60b-824e30993eae-kube-api-access-5495p\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:19:11.007542 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:11.007280 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a19a892d-c6f3-451e-b60b-824e30993eae-bundle\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:19:11.007542 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:11.007295 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3299b5b8-cd31-4b1f-9f76-dcbf5b1f5337-util\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:19:11.007542 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:11.007305 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3299b5b8-cd31-4b1f-9f76-dcbf5b1f5337-bundle\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:19:11.007542 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:11.007316 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j44p4\" (UniqueName: \"kubernetes.io/projected/3299b5b8-cd31-4b1f-9f76-dcbf5b1f5337-kube-api-access-j44p4\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:19:11.661572 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:11.661542 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pxctb" Jul 23 00:19:11.661748 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:11.661536 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pxctb" event={"ID":"a19a892d-c6f3-451e-b60b-824e30993eae","Type":"ContainerDied","Data":"91e18dd755f4304cb37314b26f0d00f73857b8cec1648e7d66d52da8ddec0612"} Jul 23 00:19:11.661748 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:11.661680 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91e18dd755f4304cb37314b26f0d00f73857b8cec1648e7d66d52da8ddec0612" Jul 23 00:19:11.663592 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:11.663564 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nm6kj" Jul 23 00:19:11.663592 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:11.663582 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nm6kj" event={"ID":"3299b5b8-cd31-4b1f-9f76-dcbf5b1f5337","Type":"ContainerDied","Data":"55896c4d2f4716f0f36260e4fe441e90bbdfd14cacb5b52155d1cae7ddccbcaa"} Jul 23 00:19:11.663779 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:19:11.663630 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55896c4d2f4716f0f36260e4fe441e90bbdfd14cacb5b52155d1cae7ddccbcaa" Jul 23 00:19:17.847246 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:19:17.847199 2562 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc5b0c3a_51dc_4e24_a6de_140c0ecb8f24.slice/crio-c7a3b81ba6450bd146a7c8bdda3522fcd7b75fb1f5d4aea49d09b21ff73c369a.scope\": RecentStats: unable to find data in memory cache]" Jul 23 00:19:17.847246 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:19:17.847221 2562 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc5b0c3a_51dc_4e24_a6de_140c0ecb8f24.slice/crio-c7a3b81ba6450bd146a7c8bdda3522fcd7b75fb1f5d4aea49d09b21ff73c369a.scope\": RecentStats: unable to find data in memory cache]" Jul 23 00:19:18.584781 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:19:18.584729 2562 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc5b0c3a_51dc_4e24_a6de_140c0ecb8f24.slice/crio-c7a3b81ba6450bd146a7c8bdda3522fcd7b75fb1f5d4aea49d09b21ff73c369a.scope\": RecentStats: unable to find data in memory cache]" Jul 23 00:19:23.703080 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:19:23.703035 2562 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc5b0c3a_51dc_4e24_a6de_140c0ecb8f24.slice/crio-c7a3b81ba6450bd146a7c8bdda3522fcd7b75fb1f5d4aea49d09b21ff73c369a.scope\": RecentStats: unable to find data in memory cache]" Jul 23 00:19:28.594204 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:19:28.594162 2562 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc5b0c3a_51dc_4e24_a6de_140c0ecb8f24.slice/crio-c7a3b81ba6450bd146a7c8bdda3522fcd7b75fb1f5d4aea49d09b21ff73c369a.scope\": RecentStats: unable to find data in memory cache]" Jul 23 00:19:38.642162 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:19:38.642118 2562 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc5b0c3a_51dc_4e24_a6de_140c0ecb8f24.slice/crio-c7a3b81ba6450bd146a7c8bdda3522fcd7b75fb1f5d4aea49d09b21ff73c369a.scope\": RecentStats: unable to find data in memory cache]" Jul 23 00:19:38.654286 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:19:38.654226 2562 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc5b0c3a_51dc_4e24_a6de_140c0ecb8f24.slice/crio-c7a3b81ba6450bd146a7c8bdda3522fcd7b75fb1f5d4aea49d09b21ff73c369a.scope\": RecentStats: unable to find data in memory cache]" Jul 23 00:19:48.688994 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:19:48.688953 2562 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc5b0c3a_51dc_4e24_a6de_140c0ecb8f24.slice/crio-c7a3b81ba6450bd146a7c8bdda3522fcd7b75fb1f5d4aea49d09b21ff73c369a.scope\": RecentStats: unable to find data in memory cache]" Jul 23 00:19:53.647620 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:19:53.647573 2562 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc5b0c3a_51dc_4e24_a6de_140c0ecb8f24.slice/crio-c7a3b81ba6450bd146a7c8bdda3522fcd7b75fb1f5d4aea49d09b21ff73c369a.scope\": RecentStats: unable to find data in memory cache]" Jul 23 00:19:58.737798 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:19:58.737684 2562 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc5b0c3a_51dc_4e24_a6de_140c0ecb8f24.slice/crio-c7a3b81ba6450bd146a7c8bdda3522fcd7b75fb1f5d4aea49d09b21ff73c369a.scope\": RecentStats: unable to find data in memory cache]" Jul 23 00:21:04.604473 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:21:04.604426 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-pvbhb"] Jul 23 00:21:04.605065 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:21:04.605042 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc5b0c3a-51dc-4e24-a6de-140c0ecb8f24" containerName="pull" Jul 23 00:21:04.605111 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:21:04.605069 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc5b0c3a-51dc-4e24-a6de-140c0ecb8f24" containerName="pull" Jul 23 00:21:04.605111 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:21:04.605083 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3299b5b8-cd31-4b1f-9f76-dcbf5b1f5337" containerName="util" Jul 23 00:21:04.605111 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:21:04.605094 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3299b5b8-cd31-4b1f-9f76-dcbf5b1f5337" containerName="util" Jul 23 00:21:04.605111 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:21:04.605108 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3299b5b8-cd31-4b1f-9f76-dcbf5b1f5337" containerName="extract" Jul 23 00:21:04.605228 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:21:04.605118 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3299b5b8-cd31-4b1f-9f76-dcbf5b1f5337" containerName="extract" Jul 23 00:21:04.605228 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:21:04.605129 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a19a892d-c6f3-451e-b60b-824e30993eae" containerName="extract" Jul 23 00:21:04.605228 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:21:04.605137 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a19a892d-c6f3-451e-b60b-824e30993eae" containerName="extract" Jul 23 00:21:04.605228 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:21:04.605158 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc5b0c3a-51dc-4e24-a6de-140c0ecb8f24" containerName="extract" Jul 23 00:21:04.605228 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:21:04.605167 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc5b0c3a-51dc-4e24-a6de-140c0ecb8f24" containerName="extract" Jul 23 00:21:04.605228 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:21:04.605186 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e929649-eea1-4da3-ab01-a77d5c4a0d34" containerName="util" Jul 23 00:21:04.605228 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:21:04.605194 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e929649-eea1-4da3-ab01-a77d5c4a0d34" containerName="util" Jul 23 00:21:04.605228 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:21:04.605204 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3299b5b8-cd31-4b1f-9f76-dcbf5b1f5337" containerName="pull" Jul 23 00:21:04.605228 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:21:04.605212 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3299b5b8-cd31-4b1f-9f76-dcbf5b1f5337" containerName="pull" Jul 23 00:21:04.605228 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:21:04.605221 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a19a892d-c6f3-451e-b60b-824e30993eae" containerName="pull" Jul 23 00:21:04.605738 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:21:04.605230 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a19a892d-c6f3-451e-b60b-824e30993eae" containerName="pull" Jul 23 00:21:04.605738 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:21:04.605245 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc5b0c3a-51dc-4e24-a6de-140c0ecb8f24" containerName="util" Jul 23 00:21:04.605738 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:21:04.605254 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc5b0c3a-51dc-4e24-a6de-140c0ecb8f24" containerName="util" Jul 23 00:21:04.605738 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:21:04.605266 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e929649-eea1-4da3-ab01-a77d5c4a0d34" containerName="pull" Jul 23 00:21:04.605738 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:21:04.605274 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e929649-eea1-4da3-ab01-a77d5c4a0d34" containerName="pull" Jul 23 00:21:04.605738 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:21:04.605284 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e929649-eea1-4da3-ab01-a77d5c4a0d34" containerName="extract" Jul 23 00:21:04.605738 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:21:04.605294 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e929649-eea1-4da3-ab01-a77d5c4a0d34" containerName="extract" Jul 23 00:21:04.605738 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:21:04.605307 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a19a892d-c6f3-451e-b60b-824e30993eae" containerName="util" Jul 23 00:21:04.605738 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:21:04.605316 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a19a892d-c6f3-451e-b60b-824e30993eae" containerName="util" Jul 23 00:21:04.605738 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:21:04.605402 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e929649-eea1-4da3-ab01-a77d5c4a0d34" containerName="extract" Jul 23 00:21:04.605738 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:21:04.605413 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc5b0c3a-51dc-4e24-a6de-140c0ecb8f24" containerName="extract" Jul 23 00:21:04.605738 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:21:04.605423 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3299b5b8-cd31-4b1f-9f76-dcbf5b1f5337" containerName="extract" Jul 23 00:21:04.605738 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:21:04.605431 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a19a892d-c6f3-451e-b60b-824e30993eae" containerName="extract" Jul 23 00:21:04.609220 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:21:04.609195 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-pvbhb" Jul 23 00:21:04.612117 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:21:04.612085 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 23 00:21:04.612271 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:21:04.612101 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 23 00:21:04.613682 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:21:04.613452 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 00:21:04.613682 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:21:04.613508 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-j75wb\"" Jul 23 00:21:04.616442 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:21:04.616413 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-pvbhb"] Jul 23 00:21:04.666077 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:21:04.666041 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fmpm\" (UniqueName: \"kubernetes.io/projected/c5796969-96a2-4b83-a70e-ebdf37f2b36e-kube-api-access-2fmpm\") pod \"postgres-69fb4594cf-pvbhb\" (UID: \"c5796969-96a2-4b83-a70e-ebdf37f2b36e\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-pvbhb" Jul 23 00:21:04.666280 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:21:04.666146 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c5796969-96a2-4b83-a70e-ebdf37f2b36e-data\") pod \"postgres-69fb4594cf-pvbhb\" (UID: \"c5796969-96a2-4b83-a70e-ebdf37f2b36e\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-pvbhb" Jul 23 00:21:04.766792 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:21:04.766723 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c5796969-96a2-4b83-a70e-ebdf37f2b36e-data\") pod \"postgres-69fb4594cf-pvbhb\" (UID: \"c5796969-96a2-4b83-a70e-ebdf37f2b36e\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-pvbhb" Jul 23 00:21:04.766968 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:21:04.766835 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fmpm\" (UniqueName: \"kubernetes.io/projected/c5796969-96a2-4b83-a70e-ebdf37f2b36e-kube-api-access-2fmpm\") pod \"postgres-69fb4594cf-pvbhb\" (UID: \"c5796969-96a2-4b83-a70e-ebdf37f2b36e\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-pvbhb" Jul 23 00:21:04.767131 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:21:04.767110 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c5796969-96a2-4b83-a70e-ebdf37f2b36e-data\") pod \"postgres-69fb4594cf-pvbhb\" (UID: \"c5796969-96a2-4b83-a70e-ebdf37f2b36e\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-pvbhb" Jul 23 00:21:04.777025 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:21:04.776992 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fmpm\" (UniqueName: \"kubernetes.io/projected/c5796969-96a2-4b83-a70e-ebdf37f2b36e-kube-api-access-2fmpm\") pod \"postgres-69fb4594cf-pvbhb\" (UID: \"c5796969-96a2-4b83-a70e-ebdf37f2b36e\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-pvbhb" Jul 23 00:21:04.922141 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:21:04.922040 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-pvbhb" Jul 23 00:21:05.065078 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:21:05.065051 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-pvbhb"] Jul 23 00:21:05.067546 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:21:05.067514 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5796969_96a2_4b83_a70e_ebdf37f2b36e.slice/crio-41c014fa53bd84a0902619d2d753c81b64bc4decce635d881a489cc0a9043b15 WatchSource:0}: Error finding container 41c014fa53bd84a0902619d2d753c81b64bc4decce635d881a489cc0a9043b15: Status 404 returned error can't find the container with id 41c014fa53bd84a0902619d2d753c81b64bc4decce635d881a489cc0a9043b15 Jul 23 00:21:05.069021 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:21:05.068998 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 00:21:05.258791 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:21:05.258675 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-pvbhb" event={"ID":"c5796969-96a2-4b83-a70e-ebdf37f2b36e","Type":"ContainerStarted","Data":"41c014fa53bd84a0902619d2d753c81b64bc4decce635d881a489cc0a9043b15"} Jul 23 00:21:10.050316 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:21:10.050290 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 23 00:21:10.293375 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:21:10.293332 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-pvbhb" event={"ID":"c5796969-96a2-4b83-a70e-ebdf37f2b36e","Type":"ContainerStarted","Data":"f56120196369e64022986e5c157790e8ce58c9d86b37874a0fba2f09bb6a2698"} Jul 23 00:21:10.293585 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:21:10.293436 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-pvbhb" Jul 23 00:21:10.310045 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:21:10.309945 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-pvbhb" podStartSLOduration=1.331410272 podStartE2EDuration="6.309930373s" totalImagesPullingTime="4.978520101s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:21:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:21:10.30852257 +0000 UTC m=+784.104029427" watchObservedRunningTime="2026-07-23 00:21:10.309930373 +0000 UTC m=+784.105437232" Jul 23 00:21:16.329952 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:21:16.329909 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-pvbhb" Jul 23 00:22:01.133502 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:22:01.133404 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-glvxf"] Jul 23 00:22:01.137342 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:22:01.137313 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-glvxf" Jul 23 00:22:01.140435 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:22:01.140411 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-6km9t\"" Jul 23 00:22:01.140435 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:22:01.140421 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 23 00:22:01.148343 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:22:01.148314 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-glvxf"] Jul 23 00:22:01.186243 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:22:01.186204 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5nhv\" (UniqueName: \"kubernetes.io/projected/baedce3c-d14a-49ac-b183-8468e48c8d89-kube-api-access-t5nhv\") pod \"payload-processing-5bf47cd988-glvxf\" (UID: \"baedce3c-d14a-49ac-b183-8468e48c8d89\") " pod="openshift-ingress/payload-processing-5bf47cd988-glvxf" Jul 23 00:22:01.186409 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:22:01.186285 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baedce3c-d14a-49ac-b183-8468e48c8d89-plugins-config-volume\") pod \"payload-processing-5bf47cd988-glvxf\" (UID: \"baedce3c-d14a-49ac-b183-8468e48c8d89\") " pod="openshift-ingress/payload-processing-5bf47cd988-glvxf" Jul 23 00:22:01.287699 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:22:01.287638 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baedce3c-d14a-49ac-b183-8468e48c8d89-plugins-config-volume\") pod \"payload-processing-5bf47cd988-glvxf\" (UID: \"baedce3c-d14a-49ac-b183-8468e48c8d89\") " pod="openshift-ingress/payload-processing-5bf47cd988-glvxf" Jul 23 00:22:01.287699 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:22:01.287710 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5nhv\" (UniqueName: \"kubernetes.io/projected/baedce3c-d14a-49ac-b183-8468e48c8d89-kube-api-access-t5nhv\") pod \"payload-processing-5bf47cd988-glvxf\" (UID: \"baedce3c-d14a-49ac-b183-8468e48c8d89\") " pod="openshift-ingress/payload-processing-5bf47cd988-glvxf" Jul 23 00:22:01.288413 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:22:01.288389 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baedce3c-d14a-49ac-b183-8468e48c8d89-plugins-config-volume\") pod \"payload-processing-5bf47cd988-glvxf\" (UID: \"baedce3c-d14a-49ac-b183-8468e48c8d89\") " pod="openshift-ingress/payload-processing-5bf47cd988-glvxf" Jul 23 00:22:01.296482 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:22:01.296454 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5nhv\" (UniqueName: \"kubernetes.io/projected/baedce3c-d14a-49ac-b183-8468e48c8d89-kube-api-access-t5nhv\") pod \"payload-processing-5bf47cd988-glvxf\" (UID: \"baedce3c-d14a-49ac-b183-8468e48c8d89\") " pod="openshift-ingress/payload-processing-5bf47cd988-glvxf" Jul 23 00:22:01.448816 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:22:01.448691 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-glvxf" Jul 23 00:22:01.589792 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:22:01.589749 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-glvxf"] Jul 23 00:22:01.592751 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:22:01.592712 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaedce3c_d14a_49ac_b183_8468e48c8d89.slice/crio-e0f683263f6895157f9734c3a42742f4f0ce7801e300dcb6069dd9f60ad15d9c WatchSource:0}: Error finding container e0f683263f6895157f9734c3a42742f4f0ce7801e300dcb6069dd9f60ad15d9c: Status 404 returned error can't find the container with id e0f683263f6895157f9734c3a42742f4f0ce7801e300dcb6069dd9f60ad15d9c Jul 23 00:22:02.583660 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:22:02.583574 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-glvxf" event={"ID":"baedce3c-d14a-49ac-b183-8468e48c8d89","Type":"ContainerStarted","Data":"e0f683263f6895157f9734c3a42742f4f0ce7801e300dcb6069dd9f60ad15d9c"} Jul 23 00:22:04.595864 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:22:04.595825 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-glvxf" event={"ID":"baedce3c-d14a-49ac-b183-8468e48c8d89","Type":"ContainerStarted","Data":"774f29ead579e87e5e738f0620e2168dab3b432a9fd8d2c6f07965e1929f1b53"} Jul 23 00:22:04.596279 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:22:04.595960 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5bf47cd988-glvxf" Jul 23 00:22:04.612388 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:22:04.612329 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5bf47cd988-glvxf" podStartSLOduration=1.6828201809999999 podStartE2EDuration="3.612313056s" totalImagesPullingTime="1.929492875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:22:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:22:04.611095609 +0000 UTC m=+838.406602467" watchObservedRunningTime="2026-07-23 00:22:04.612313056 +0000 UTC m=+838.407819913" Jul 23 00:22:15.604957 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:22:15.604921 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5bf47cd988-glvxf" Jul 23 00:22:45.721104 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:22:45.721064 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-b7w99"] Jul 23 00:22:45.725317 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:22:45.725291 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b7w99" Jul 23 00:22:45.728277 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:22:45.728246 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 23 00:22:45.728412 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:22:45.728380 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 23 00:22:45.729722 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:22:45.729584 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-jghlj\"" Jul 23 00:22:45.730383 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:22:45.730359 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 23 00:22:45.733961 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:22:45.733932 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-b7w99"] Jul 23 00:22:45.798531 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:22:45.798498 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5635ea5d-e3cc-46fe-b8dd-e498b18018fe-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-b7w99\" (UID: \"5635ea5d-e3cc-46fe-b8dd-e498b18018fe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b7w99" Jul 23 00:22:45.798710 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:22:45.798542 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpgm6\" (UniqueName: \"kubernetes.io/projected/5635ea5d-e3cc-46fe-b8dd-e498b18018fe-kube-api-access-hpgm6\") pod \"e2e-trlp-test-simulated-kserve-598466548d-b7w99\" (UID: \"5635ea5d-e3cc-46fe-b8dd-e498b18018fe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b7w99" Jul 23 00:22:45.798710 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:22:45.798584 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5635ea5d-e3cc-46fe-b8dd-e498b18018fe-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-b7w99\" (UID: \"5635ea5d-e3cc-46fe-b8dd-e498b18018fe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b7w99" Jul 23 00:22:45.798710 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:22:45.798639 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5635ea5d-e3cc-46fe-b8dd-e498b18018fe-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-b7w99\" (UID: \"5635ea5d-e3cc-46fe-b8dd-e498b18018fe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b7w99" Jul 23 00:22:45.798710 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:22:45.798675 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5635ea5d-e3cc-46fe-b8dd-e498b18018fe-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-b7w99\" (UID: \"5635ea5d-e3cc-46fe-b8dd-e498b18018fe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b7w99" Jul 23 00:22:45.798871 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:22:45.798744 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5635ea5d-e3cc-46fe-b8dd-e498b18018fe-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-b7w99\" (UID: \"5635ea5d-e3cc-46fe-b8dd-e498b18018fe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b7w99" Jul 23 00:22:45.899854 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:22:45.899795 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5635ea5d-e3cc-46fe-b8dd-e498b18018fe-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-b7w99\" (UID: \"5635ea5d-e3cc-46fe-b8dd-e498b18018fe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b7w99" Jul 23 00:22:45.899854 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:22:45.899861 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5635ea5d-e3cc-46fe-b8dd-e498b18018fe-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-b7w99\" (UID: \"5635ea5d-e3cc-46fe-b8dd-e498b18018fe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b7w99" Jul 23 00:22:45.900132 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:22:45.899915 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5635ea5d-e3cc-46fe-b8dd-e498b18018fe-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-b7w99\" (UID: \"5635ea5d-e3cc-46fe-b8dd-e498b18018fe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b7w99" Jul 23 00:22:45.900132 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:22:45.899935 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5635ea5d-e3cc-46fe-b8dd-e498b18018fe-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-b7w99\" (UID: \"5635ea5d-e3cc-46fe-b8dd-e498b18018fe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b7w99" Jul 23 00:22:45.900132 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:22:45.899959 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpgm6\" (UniqueName: \"kubernetes.io/projected/5635ea5d-e3cc-46fe-b8dd-e498b18018fe-kube-api-access-hpgm6\") pod \"e2e-trlp-test-simulated-kserve-598466548d-b7w99\" (UID: \"5635ea5d-e3cc-46fe-b8dd-e498b18018fe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b7w99" Jul 23 00:22:45.900132 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:22:45.900072 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5635ea5d-e3cc-46fe-b8dd-e498b18018fe-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-b7w99\" (UID: \"5635ea5d-e3cc-46fe-b8dd-e498b18018fe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b7w99" Jul 23 00:22:45.900345 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:22:45.900321 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5635ea5d-e3cc-46fe-b8dd-e498b18018fe-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-b7w99\" (UID: \"5635ea5d-e3cc-46fe-b8dd-e498b18018fe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b7w99" Jul 23 00:22:45.900393 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:22:45.900382 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5635ea5d-e3cc-46fe-b8dd-e498b18018fe-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-b7w99\" (UID: \"5635ea5d-e3cc-46fe-b8dd-e498b18018fe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b7w99" Jul 23 00:22:45.900490 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:22:45.900463 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5635ea5d-e3cc-46fe-b8dd-e498b18018fe-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-b7w99\" (UID: \"5635ea5d-e3cc-46fe-b8dd-e498b18018fe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b7w99" Jul 23 00:22:45.902283 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:22:45.902252 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5635ea5d-e3cc-46fe-b8dd-e498b18018fe-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-b7w99\" (UID: \"5635ea5d-e3cc-46fe-b8dd-e498b18018fe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b7w99" Jul 23 00:22:45.902563 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:22:45.902544 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5635ea5d-e3cc-46fe-b8dd-e498b18018fe-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-b7w99\" (UID: \"5635ea5d-e3cc-46fe-b8dd-e498b18018fe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b7w99" Jul 23 00:22:45.908530 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:22:45.908504 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpgm6\" (UniqueName: \"kubernetes.io/projected/5635ea5d-e3cc-46fe-b8dd-e498b18018fe-kube-api-access-hpgm6\") pod \"e2e-trlp-test-simulated-kserve-598466548d-b7w99\" (UID: \"5635ea5d-e3cc-46fe-b8dd-e498b18018fe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b7w99" Jul 23 00:22:46.038211 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:22:46.038164 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b7w99" Jul 23 00:22:46.181428 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:22:46.181402 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-b7w99"] Jul 23 00:22:46.182803 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:22:46.182767 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5635ea5d_e3cc_46fe_b8dd_e498b18018fe.slice/crio-cdcc210417e4e66a86ddf034f61b4f984895db08d6868f71ac1f6cd180413300 WatchSource:0}: Error finding container cdcc210417e4e66a86ddf034f61b4f984895db08d6868f71ac1f6cd180413300: Status 404 returned error can't find the container with id cdcc210417e4e66a86ddf034f61b4f984895db08d6868f71ac1f6cd180413300 Jul 23 00:22:46.826398 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:22:46.826354 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b7w99" event={"ID":"5635ea5d-e3cc-46fe-b8dd-e498b18018fe","Type":"ContainerStarted","Data":"cdcc210417e4e66a86ddf034f61b4f984895db08d6868f71ac1f6cd180413300"} Jul 23 00:22:47.833459 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:22:47.833412 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b7w99" event={"ID":"5635ea5d-e3cc-46fe-b8dd-e498b18018fe","Type":"ContainerStarted","Data":"6eb14ffdb8a3fba2ed984758432944e6e31d19134645509c3ff2591d6bb42d6c"} Jul 23 00:22:47.857966 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:22:47.855845 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b7w99" podStartSLOduration=1.8372921880000002 podStartE2EDuration="2.855816261s" totalImagesPullingTime="1.018524073s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:22:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:22:47.851435463 +0000 UTC m=+881.646942322" watchObservedRunningTime="2026-07-23 00:22:47.855816261 +0000 UTC m=+881.651323119" Jul 23 00:22:56.038947 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:22:56.038905 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b7w99" Jul 23 00:22:56.039579 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:22:56.038979 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b7w99" Jul 23 00:22:56.052227 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:22:56.052189 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b7w99" Jul 23 00:22:56.892341 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:22:56.892307 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-b7w99" Jul 23 00:30:00.141558 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:30:00.141516 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29746110-jbxnb"] Jul 23 00:30:00.145019 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:30:00.144997 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746110-jbxnb" Jul 23 00:30:00.147835 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:30:00.147813 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w8tk2\"" Jul 23 00:30:00.155570 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:30:00.155528 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746110-jbxnb" Jul 23 00:30:00.159502 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:30:00.158894 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29746110-jbxnb"] Jul 23 00:30:00.301471 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:30:00.301445 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29746110-jbxnb"] Jul 23 00:30:00.303529 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:30:00.303495 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26fd18d8_a27c_4ad6_8e87_7dfe920bd797.slice/crio-010d7b5af1ef94f859bdcf1b10c42c57a329fb6dc9407e4dda0aeaa1daa94818 WatchSource:0}: Error finding container 010d7b5af1ef94f859bdcf1b10c42c57a329fb6dc9407e4dda0aeaa1daa94818: Status 404 returned error can't find the container with id 010d7b5af1ef94f859bdcf1b10c42c57a329fb6dc9407e4dda0aeaa1daa94818 Jul 23 00:30:00.305440 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:30:00.305423 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 00:30:01.160087 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:30:01.160047 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746110-jbxnb" event={"ID":"26fd18d8-a27c-4ad6-8e87-7dfe920bd797","Type":"ContainerStarted","Data":"010d7b5af1ef94f859bdcf1b10c42c57a329fb6dc9407e4dda0aeaa1daa94818"} Jul 23 00:30:03.173124 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:30:03.173073 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746110-jbxnb" event={"ID":"26fd18d8-a27c-4ad6-8e87-7dfe920bd797","Type":"ContainerStarted","Data":"511d6ef0dd2cfc4bb3c82ee17a9be4698d6f3aa0eb90fd05632a2b6f4e9a45dd"} Jul 23 00:30:03.188111 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:30:03.188019 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746110-jbxnb" podStartSLOduration=1.02273568 podStartE2EDuration="3.187998545s" totalImagesPullingTime="2.165262865s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:30:03.187391631 +0000 UTC m=+1316.982898488" watchObservedRunningTime="2026-07-23 00:30:03.187998545 +0000 UTC m=+1316.983505403" Jul 23 00:30:04.180165 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:30:04.180128 2562 generic.go:356] "Generic (PLEG): container finished" podID="26fd18d8-a27c-4ad6-8e87-7dfe920bd797" containerID="511d6ef0dd2cfc4bb3c82ee17a9be4698d6f3aa0eb90fd05632a2b6f4e9a45dd" exitCode=0 Jul 23 00:30:04.180639 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:30:04.180208 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746110-jbxnb" event={"ID":"26fd18d8-a27c-4ad6-8e87-7dfe920bd797","Type":"ContainerDied","Data":"511d6ef0dd2cfc4bb3c82ee17a9be4698d6f3aa0eb90fd05632a2b6f4e9a45dd"} Jul 23 00:30:05.327164 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:30:05.327136 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746110-jbxnb" Jul 23 00:30:06.191712 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:30:06.191677 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746110-jbxnb" event={"ID":"26fd18d8-a27c-4ad6-8e87-7dfe920bd797","Type":"ContainerDied","Data":"010d7b5af1ef94f859bdcf1b10c42c57a329fb6dc9407e4dda0aeaa1daa94818"} Jul 23 00:30:06.191712 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:30:06.191694 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746110-jbxnb" Jul 23 00:30:06.191989 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:30:06.191729 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="010d7b5af1ef94f859bdcf1b10c42c57a329fb6dc9407e4dda0aeaa1daa94818" Jul 23 00:40:12.620214 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:40:12.620179 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw"] Jul 23 00:40:12.620691 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:40:12.620557 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26fd18d8-a27c-4ad6-8e87-7dfe920bd797" containerName="cleanup" Jul 23 00:40:12.620691 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:40:12.620569 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="26fd18d8-a27c-4ad6-8e87-7dfe920bd797" containerName="cleanup" Jul 23 00:40:12.620691 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:40:12.620646 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="26fd18d8-a27c-4ad6-8e87-7dfe920bd797" containerName="cleanup" Jul 23 00:40:12.623724 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:40:12.623703 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw" Jul 23 00:40:12.626492 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:40:12.626474 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-3eadb28c-openshift-default-dockercfg-kcx4j\"" Jul 23 00:40:12.647192 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:40:12.647158 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw"] Jul 23 00:40:12.695299 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:40:12.695252 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/daf9c197-91ee-442f-9c32-e257835d26d2-workload-certs\") pod \"e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw\" (UID: \"daf9c197-91ee-442f-9c32-e257835d26d2\") " pod="openshift-ingress/e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw" Jul 23 00:40:12.695299 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:40:12.695298 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/daf9c197-91ee-442f-9c32-e257835d26d2-istiod-ca-cert\") pod \"e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw\" (UID: \"daf9c197-91ee-442f-9c32-e257835d26d2\") " pod="openshift-ingress/e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw" Jul 23 00:40:12.695533 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:40:12.695385 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/daf9c197-91ee-442f-9c32-e257835d26d2-istio-token\") pod \"e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw\" (UID: \"daf9c197-91ee-442f-9c32-e257835d26d2\") " pod="openshift-ingress/e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw" Jul 23 00:40:12.695533 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:40:12.695470 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/daf9c197-91ee-442f-9c32-e257835d26d2-workload-socket\") pod \"e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw\" (UID: \"daf9c197-91ee-442f-9c32-e257835d26d2\") " pod="openshift-ingress/e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw" Jul 23 00:40:12.695533 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:40:12.695522 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/daf9c197-91ee-442f-9c32-e257835d26d2-istio-podinfo\") pod \"e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw\" (UID: \"daf9c197-91ee-442f-9c32-e257835d26d2\") " pod="openshift-ingress/e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw" Jul 23 00:40:12.695695 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:40:12.695597 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/daf9c197-91ee-442f-9c32-e257835d26d2-credential-socket\") pod \"e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw\" (UID: \"daf9c197-91ee-442f-9c32-e257835d26d2\") " pod="openshift-ingress/e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw" Jul 23 00:40:12.695695 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:40:12.695683 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/daf9c197-91ee-442f-9c32-e257835d26d2-istio-envoy\") pod \"e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw\" (UID: \"daf9c197-91ee-442f-9c32-e257835d26d2\") " pod="openshift-ingress/e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw" Jul 23 00:40:12.695853 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:40:12.695720 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqx4q\" (UniqueName: \"kubernetes.io/projected/daf9c197-91ee-442f-9c32-e257835d26d2-kube-api-access-bqx4q\") pod \"e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw\" (UID: \"daf9c197-91ee-442f-9c32-e257835d26d2\") " pod="openshift-ingress/e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw" Jul 23 00:40:12.695853 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:40:12.695750 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/daf9c197-91ee-442f-9c32-e257835d26d2-istio-data\") pod \"e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw\" (UID: \"daf9c197-91ee-442f-9c32-e257835d26d2\") " pod="openshift-ingress/e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw" Jul 23 00:40:12.796772 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:40:12.796719 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/daf9c197-91ee-442f-9c32-e257835d26d2-istio-envoy\") pod \"e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw\" (UID: \"daf9c197-91ee-442f-9c32-e257835d26d2\") " pod="openshift-ingress/e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw" Jul 23 00:40:12.796970 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:40:12.796804 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqx4q\" (UniqueName: \"kubernetes.io/projected/daf9c197-91ee-442f-9c32-e257835d26d2-kube-api-access-bqx4q\") pod \"e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw\" (UID: \"daf9c197-91ee-442f-9c32-e257835d26d2\") " pod="openshift-ingress/e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw" Jul 23 00:40:12.796970 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:40:12.796840 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/daf9c197-91ee-442f-9c32-e257835d26d2-istio-data\") pod \"e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw\" (UID: \"daf9c197-91ee-442f-9c32-e257835d26d2\") " pod="openshift-ingress/e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw" Jul 23 00:40:12.796970 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:40:12.796884 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/daf9c197-91ee-442f-9c32-e257835d26d2-workload-certs\") pod \"e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw\" (UID: \"daf9c197-91ee-442f-9c32-e257835d26d2\") " pod="openshift-ingress/e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw" Jul 23 00:40:12.796970 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:40:12.796909 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/daf9c197-91ee-442f-9c32-e257835d26d2-istiod-ca-cert\") pod \"e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw\" (UID: \"daf9c197-91ee-442f-9c32-e257835d26d2\") " pod="openshift-ingress/e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw" Jul 23 00:40:12.796970 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:40:12.796954 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/daf9c197-91ee-442f-9c32-e257835d26d2-istio-token\") pod \"e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw\" (UID: \"daf9c197-91ee-442f-9c32-e257835d26d2\") " pod="openshift-ingress/e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw" Jul 23 00:40:12.797227 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:40:12.797055 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/daf9c197-91ee-442f-9c32-e257835d26d2-workload-socket\") pod \"e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw\" (UID: \"daf9c197-91ee-442f-9c32-e257835d26d2\") " pod="openshift-ingress/e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw" Jul 23 00:40:12.797227 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:40:12.797123 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/daf9c197-91ee-442f-9c32-e257835d26d2-istio-podinfo\") pod \"e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw\" (UID: \"daf9c197-91ee-442f-9c32-e257835d26d2\") " pod="openshift-ingress/e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw" Jul 23 00:40:12.797227 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:40:12.797163 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/daf9c197-91ee-442f-9c32-e257835d26d2-credential-socket\") pod \"e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw\" (UID: \"daf9c197-91ee-442f-9c32-e257835d26d2\") " pod="openshift-ingress/e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw" Jul 23 00:40:12.797385 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:40:12.797343 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/daf9c197-91ee-442f-9c32-e257835d26d2-istio-data\") pod \"e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw\" (UID: \"daf9c197-91ee-442f-9c32-e257835d26d2\") " pod="openshift-ingress/e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw" Jul 23 00:40:12.797443 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:40:12.797403 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/daf9c197-91ee-442f-9c32-e257835d26d2-workload-socket\") pod \"e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw\" (UID: \"daf9c197-91ee-442f-9c32-e257835d26d2\") " pod="openshift-ingress/e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw" Jul 23 00:40:12.797443 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:40:12.797425 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/daf9c197-91ee-442f-9c32-e257835d26d2-workload-certs\") pod \"e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw\" (UID: \"daf9c197-91ee-442f-9c32-e257835d26d2\") " pod="openshift-ingress/e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw" Jul 23 00:40:12.797550 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:40:12.797494 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/daf9c197-91ee-442f-9c32-e257835d26d2-credential-socket\") pod \"e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw\" (UID: \"daf9c197-91ee-442f-9c32-e257835d26d2\") " pod="openshift-ingress/e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw" Jul 23 00:40:12.797889 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:40:12.797864 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/daf9c197-91ee-442f-9c32-e257835d26d2-istiod-ca-cert\") pod \"e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw\" (UID: \"daf9c197-91ee-442f-9c32-e257835d26d2\") " pod="openshift-ingress/e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw" Jul 23 00:40:12.799226 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:40:12.799205 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/daf9c197-91ee-442f-9c32-e257835d26d2-istio-envoy\") pod \"e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw\" (UID: \"daf9c197-91ee-442f-9c32-e257835d26d2\") " pod="openshift-ingress/e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw" Jul 23 00:40:12.799597 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:40:12.799576 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/daf9c197-91ee-442f-9c32-e257835d26d2-istio-podinfo\") pod \"e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw\" (UID: \"daf9c197-91ee-442f-9c32-e257835d26d2\") " pod="openshift-ingress/e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw" Jul 23 00:40:12.805229 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:40:12.805206 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/daf9c197-91ee-442f-9c32-e257835d26d2-istio-token\") pod \"e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw\" (UID: \"daf9c197-91ee-442f-9c32-e257835d26d2\") " pod="openshift-ingress/e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw" Jul 23 00:40:12.805368 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:40:12.805352 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqx4q\" (UniqueName: \"kubernetes.io/projected/daf9c197-91ee-442f-9c32-e257835d26d2-kube-api-access-bqx4q\") pod \"e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw\" (UID: \"daf9c197-91ee-442f-9c32-e257835d26d2\") " pod="openshift-ingress/e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw" Jul 23 00:40:12.933780 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:40:12.933685 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw" Jul 23 00:40:13.078527 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:40:13.078500 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw"] Jul 23 00:40:13.080923 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:40:13.080899 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaf9c197_91ee_442f_9c32_e257835d26d2.slice/crio-b8a8dbe7cd005b60adfb9e3d2b9505b9dde95f2a50afd7a832d52f4ed358cc31 WatchSource:0}: Error finding container b8a8dbe7cd005b60adfb9e3d2b9505b9dde95f2a50afd7a832d52f4ed358cc31: Status 404 returned error can't find the container with id b8a8dbe7cd005b60adfb9e3d2b9505b9dde95f2a50afd7a832d52f4ed358cc31 Jul 23 00:40:13.083231 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:40:13.083196 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:40:13.083322 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:40:13.083262 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:40:13.083322 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:40:13.083303 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:40:13.541323 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:40:13.541280 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw" event={"ID":"daf9c197-91ee-442f-9c32-e257835d26d2","Type":"ContainerStarted","Data":"75ccdbdd4d292a01d32e42d8690b2a4411efaf9e9464dbd92eb1d58c369a304a"} Jul 23 00:40:13.541537 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:40:13.541344 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw" event={"ID":"daf9c197-91ee-442f-9c32-e257835d26d2","Type":"ContainerStarted","Data":"b8a8dbe7cd005b60adfb9e3d2b9505b9dde95f2a50afd7a832d52f4ed358cc31"} Jul 23 00:40:13.565696 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:40:13.565632 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw" podStartSLOduration=1.565617434 podStartE2EDuration="1.565617434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:40:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:40:13.563122439 +0000 UTC m=+1927.358629296" watchObservedRunningTime="2026-07-23 00:40:13.565617434 +0000 UTC m=+1927.361124290" Jul 23 00:40:13.934242 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:40:13.934133 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw" Jul 23 00:40:14.939332 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:40:14.939299 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw" Jul 23 00:40:15.552688 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:40:15.552651 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw" Jul 23 00:40:15.553750 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:40:15.553729 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw" Jul 23 00:40:31.593984 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:40:31.593929 2562 patch_prober.go:28] interesting pod/e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 23 00:40:31.594425 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:40:31.594037 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw" podUID="daf9c197-91ee-442f-9c32-e257835d26d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 23 00:40:32.657449 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:40:32.657412 2562 generic.go:356] "Generic (PLEG): container finished" podID="daf9c197-91ee-442f-9c32-e257835d26d2" containerID="75ccdbdd4d292a01d32e42d8690b2a4411efaf9e9464dbd92eb1d58c369a304a" exitCode=0 Jul 23 00:40:32.657922 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:40:32.657484 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw" event={"ID":"daf9c197-91ee-442f-9c32-e257835d26d2","Type":"ContainerDied","Data":"75ccdbdd4d292a01d32e42d8690b2a4411efaf9e9464dbd92eb1d58c369a304a"} Jul 23 00:40:32.657922 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:40:32.657901 2562 scope.go:117] "RemoveContainer" containerID="75ccdbdd4d292a01d32e42d8690b2a4411efaf9e9464dbd92eb1d58c369a304a" Jul 23 00:40:32.658888 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:40:32.658862 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:40:32.658958 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:40:32.658945 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:40:32.658997 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:40:32.658988 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:40:33.665416 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:40:33.665374 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw" event={"ID":"daf9c197-91ee-442f-9c32-e257835d26d2","Type":"ContainerStarted","Data":"0421677fee3a2c5cdc857fdac629c76a09e310d02d447275e6a909cedb8a642b"} Jul 23 00:40:33.934099 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:40:33.933999 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw" Jul 23 00:40:33.934297 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:40:33.934271 2562 patch_prober.go:28] interesting pod/e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 23 00:40:33.934370 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:40:33.934332 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw" podUID="daf9c197-91ee-442f-9c32-e257835d26d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 23 00:40:34.934361 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:40:34.934316 2562 patch_prober.go:28] interesting pod/e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 23 00:40:34.934822 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:40:34.934393 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw" podUID="daf9c197-91ee-442f-9c32-e257835d26d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 23 00:40:35.934679 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:40:35.934641 2562 patch_prober.go:28] interesting pod/e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 23 00:40:35.935205 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:40:35.934709 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw" podUID="daf9c197-91ee-442f-9c32-e257835d26d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 23 00:40:36.934461 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:40:36.934418 2562 patch_prober.go:28] interesting pod/e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 23 00:40:36.934649 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:40:36.934483 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw" podUID="daf9c197-91ee-442f-9c32-e257835d26d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 23 00:40:37.938418 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:40:37.938381 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw" Jul 23 00:40:37.938891 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:40:37.938659 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw" Jul 23 00:40:37.939328 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:40:37.939311 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw" Jul 23 00:41:05.782245 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:05.782146 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw"] Jul 23 00:41:05.782808 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:05.782424 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw" podUID="daf9c197-91ee-442f-9c32-e257835d26d2" containerName="istio-proxy" containerID="cri-o://0421677fee3a2c5cdc857fdac629c76a09e310d02d447275e6a909cedb8a642b" gracePeriod=30 Jul 23 00:41:07.941223 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:07.941183 2562 patch_prober.go:28] interesting pod/e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 23 00:41:07.941686 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:07.941248 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw" podUID="daf9c197-91ee-442f-9c32-e257835d26d2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 23 00:41:11.032585 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:11.032552 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw" Jul 23 00:41:11.168948 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:11.168843 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/daf9c197-91ee-442f-9c32-e257835d26d2-istiod-ca-cert\") pod \"daf9c197-91ee-442f-9c32-e257835d26d2\" (UID: \"daf9c197-91ee-442f-9c32-e257835d26d2\") " Jul 23 00:41:11.168948 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:11.168895 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/daf9c197-91ee-442f-9c32-e257835d26d2-istio-token\") pod \"daf9c197-91ee-442f-9c32-e257835d26d2\" (UID: \"daf9c197-91ee-442f-9c32-e257835d26d2\") " Jul 23 00:41:11.168948 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:11.168947 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/daf9c197-91ee-442f-9c32-e257835d26d2-istio-podinfo\") pod \"daf9c197-91ee-442f-9c32-e257835d26d2\" (UID: \"daf9c197-91ee-442f-9c32-e257835d26d2\") " Jul 23 00:41:11.169276 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:11.168963 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/daf9c197-91ee-442f-9c32-e257835d26d2-istio-envoy\") pod \"daf9c197-91ee-442f-9c32-e257835d26d2\" (UID: \"daf9c197-91ee-442f-9c32-e257835d26d2\") " Jul 23 00:41:11.169276 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:11.169009 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/daf9c197-91ee-442f-9c32-e257835d26d2-workload-socket\") pod \"daf9c197-91ee-442f-9c32-e257835d26d2\" (UID: \"daf9c197-91ee-442f-9c32-e257835d26d2\") " Jul 23 00:41:11.169276 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:11.169047 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/daf9c197-91ee-442f-9c32-e257835d26d2-credential-socket\") pod \"daf9c197-91ee-442f-9c32-e257835d26d2\" (UID: \"daf9c197-91ee-442f-9c32-e257835d26d2\") " Jul 23 00:41:11.169276 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:11.169071 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/daf9c197-91ee-442f-9c32-e257835d26d2-istio-data\") pod \"daf9c197-91ee-442f-9c32-e257835d26d2\" (UID: \"daf9c197-91ee-442f-9c32-e257835d26d2\") " Jul 23 00:41:11.169276 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:11.169099 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/daf9c197-91ee-442f-9c32-e257835d26d2-workload-certs\") pod \"daf9c197-91ee-442f-9c32-e257835d26d2\" (UID: \"daf9c197-91ee-442f-9c32-e257835d26d2\") " Jul 23 00:41:11.169276 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:11.169149 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bqx4q\" (UniqueName: \"kubernetes.io/projected/daf9c197-91ee-442f-9c32-e257835d26d2-kube-api-access-bqx4q\") pod \"daf9c197-91ee-442f-9c32-e257835d26d2\" (UID: \"daf9c197-91ee-442f-9c32-e257835d26d2\") " Jul 23 00:41:11.169579 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:11.169290 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daf9c197-91ee-442f-9c32-e257835d26d2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "daf9c197-91ee-442f-9c32-e257835d26d2" (UID: "daf9c197-91ee-442f-9c32-e257835d26d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:41:11.169579 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:11.169410 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daf9c197-91ee-442f-9c32-e257835d26d2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "daf9c197-91ee-442f-9c32-e257835d26d2" (UID: "daf9c197-91ee-442f-9c32-e257835d26d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:41:11.169579 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:11.169391 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daf9c197-91ee-442f-9c32-e257835d26d2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "daf9c197-91ee-442f-9c32-e257835d26d2" (UID: "daf9c197-91ee-442f-9c32-e257835d26d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:41:11.169579 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:11.169517 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/daf9c197-91ee-442f-9c32-e257835d26d2-workload-socket\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:41:11.169579 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:11.169549 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/daf9c197-91ee-442f-9c32-e257835d26d2-credential-socket\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:41:11.169579 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:11.169563 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/daf9c197-91ee-442f-9c32-e257835d26d2-istiod-ca-cert\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:41:11.169579 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:11.169577 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daf9c197-91ee-442f-9c32-e257835d26d2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "daf9c197-91ee-442f-9c32-e257835d26d2" (UID: "daf9c197-91ee-442f-9c32-e257835d26d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:41:11.170064 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:11.170037 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daf9c197-91ee-442f-9c32-e257835d26d2-istio-data" (OuterVolumeSpecName: "istio-data") pod "daf9c197-91ee-442f-9c32-e257835d26d2" (UID: "daf9c197-91ee-442f-9c32-e257835d26d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:41:11.171545 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:11.171508 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/daf9c197-91ee-442f-9c32-e257835d26d2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "daf9c197-91ee-442f-9c32-e257835d26d2" (UID: "daf9c197-91ee-442f-9c32-e257835d26d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 00:41:11.171648 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:11.171538 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daf9c197-91ee-442f-9c32-e257835d26d2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "daf9c197-91ee-442f-9c32-e257835d26d2" (UID: "daf9c197-91ee-442f-9c32-e257835d26d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:41:11.171839 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:11.171817 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daf9c197-91ee-442f-9c32-e257835d26d2-istio-token" (OuterVolumeSpecName: "istio-token") pod "daf9c197-91ee-442f-9c32-e257835d26d2" (UID: "daf9c197-91ee-442f-9c32-e257835d26d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:41:11.171919 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:11.171837 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daf9c197-91ee-442f-9c32-e257835d26d2-kube-api-access-bqx4q" (OuterVolumeSpecName: "kube-api-access-bqx4q") pod "daf9c197-91ee-442f-9c32-e257835d26d2" (UID: "daf9c197-91ee-442f-9c32-e257835d26d2"). InnerVolumeSpecName "kube-api-access-bqx4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:41:11.270250 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:11.270202 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/daf9c197-91ee-442f-9c32-e257835d26d2-istio-data\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:41:11.270250 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:11.270242 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/daf9c197-91ee-442f-9c32-e257835d26d2-workload-certs\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:41:11.270250 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:11.270258 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqx4q\" (UniqueName: \"kubernetes.io/projected/daf9c197-91ee-442f-9c32-e257835d26d2-kube-api-access-bqx4q\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:41:11.270514 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:11.270272 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/daf9c197-91ee-442f-9c32-e257835d26d2-istio-token\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:41:11.270514 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:11.270283 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/daf9c197-91ee-442f-9c32-e257835d26d2-istio-podinfo\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:41:11.270514 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:11.270295 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/daf9c197-91ee-442f-9c32-e257835d26d2-istio-envoy\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:41:11.875158 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:11.875116 2562 generic.go:356] "Generic (PLEG): container finished" podID="daf9c197-91ee-442f-9c32-e257835d26d2" containerID="0421677fee3a2c5cdc857fdac629c76a09e310d02d447275e6a909cedb8a642b" exitCode=0 Jul 23 00:41:11.875342 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:11.875165 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw" event={"ID":"daf9c197-91ee-442f-9c32-e257835d26d2","Type":"ContainerDied","Data":"0421677fee3a2c5cdc857fdac629c76a09e310d02d447275e6a909cedb8a642b"} Jul 23 00:41:11.875342 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:11.875178 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw" Jul 23 00:41:11.875342 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:11.875235 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw" event={"ID":"daf9c197-91ee-442f-9c32-e257835d26d2","Type":"ContainerDied","Data":"b8a8dbe7cd005b60adfb9e3d2b9505b9dde95f2a50afd7a832d52f4ed358cc31"} Jul 23 00:41:11.875342 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:11.875258 2562 scope.go:117] "RemoveContainer" containerID="0421677fee3a2c5cdc857fdac629c76a09e310d02d447275e6a909cedb8a642b" Jul 23 00:41:11.886013 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:11.885983 2562 scope.go:117] "RemoveContainer" containerID="75ccdbdd4d292a01d32e42d8690b2a4411efaf9e9464dbd92eb1d58c369a304a" Jul 23 00:41:11.895507 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:11.895266 2562 scope.go:117] "RemoveContainer" containerID="0421677fee3a2c5cdc857fdac629c76a09e310d02d447275e6a909cedb8a642b" Jul 23 00:41:11.895704 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:41:11.895669 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0421677fee3a2c5cdc857fdac629c76a09e310d02d447275e6a909cedb8a642b\": container with ID starting with 0421677fee3a2c5cdc857fdac629c76a09e310d02d447275e6a909cedb8a642b not found: ID does not exist" containerID="0421677fee3a2c5cdc857fdac629c76a09e310d02d447275e6a909cedb8a642b" Jul 23 00:41:11.895782 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:11.895715 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0421677fee3a2c5cdc857fdac629c76a09e310d02d447275e6a909cedb8a642b"} err="failed to get container status \"0421677fee3a2c5cdc857fdac629c76a09e310d02d447275e6a909cedb8a642b\": rpc error: code = NotFound desc = could not find container \"0421677fee3a2c5cdc857fdac629c76a09e310d02d447275e6a909cedb8a642b\": container with ID starting with 0421677fee3a2c5cdc857fdac629c76a09e310d02d447275e6a909cedb8a642b not found: ID does not exist" Jul 23 00:41:11.895782 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:11.895742 2562 scope.go:117] "RemoveContainer" containerID="75ccdbdd4d292a01d32e42d8690b2a4411efaf9e9464dbd92eb1d58c369a304a" Jul 23 00:41:11.896181 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:41:11.896154 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75ccdbdd4d292a01d32e42d8690b2a4411efaf9e9464dbd92eb1d58c369a304a\": container with ID starting with 75ccdbdd4d292a01d32e42d8690b2a4411efaf9e9464dbd92eb1d58c369a304a not found: ID does not exist" containerID="75ccdbdd4d292a01d32e42d8690b2a4411efaf9e9464dbd92eb1d58c369a304a" Jul 23 00:41:11.896300 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:11.896188 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75ccdbdd4d292a01d32e42d8690b2a4411efaf9e9464dbd92eb1d58c369a304a"} err="failed to get container status \"75ccdbdd4d292a01d32e42d8690b2a4411efaf9e9464dbd92eb1d58c369a304a\": rpc error: code = NotFound desc = could not find container \"75ccdbdd4d292a01d32e42d8690b2a4411efaf9e9464dbd92eb1d58c369a304a\": container with ID starting with 75ccdbdd4d292a01d32e42d8690b2a4411efaf9e9464dbd92eb1d58c369a304a not found: ID does not exist" Jul 23 00:41:11.899435 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:11.899346 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw"] Jul 23 00:41:11.901855 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:11.901752 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-3eadb28c-openshift-default-59ddd6955-fqvpw"] Jul 23 00:41:13.008241 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:13.008202 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="daf9c197-91ee-442f-9c32-e257835d26d2" path="/var/lib/kubelet/pods/daf9c197-91ee-442f-9c32-e257835d26d2/volumes" Jul 23 00:41:33.919785 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:33.919728 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-72954db1-727gx"] Jul 23 00:41:33.920217 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:33.920199 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="daf9c197-91ee-442f-9c32-e257835d26d2" containerName="istio-proxy" Jul 23 00:41:33.920217 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:33.920213 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="daf9c197-91ee-442f-9c32-e257835d26d2" containerName="istio-proxy" Jul 23 00:41:33.920306 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:33.920226 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="daf9c197-91ee-442f-9c32-e257835d26d2" containerName="istio-proxy" Jul 23 00:41:33.920306 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:33.920232 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="daf9c197-91ee-442f-9c32-e257835d26d2" containerName="istio-proxy" Jul 23 00:41:33.920306 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:33.920295 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="daf9c197-91ee-442f-9c32-e257835d26d2" containerName="istio-proxy" Jul 23 00:41:33.923379 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:33.923361 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-72954db1-727gx" Jul 23 00:41:33.926584 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:33.926554 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w8tk2\"" Jul 23 00:41:33.929916 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:33.929888 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-72954db1-727gx"] Jul 23 00:41:33.973382 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:33.973337 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27ef8f56-88d3-4b6e-9014-ff92c20bab2f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-72954db1-727gx\" (UID: \"27ef8f56-88d3-4b6e-9014-ff92c20bab2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-72954db1-727gx" Jul 23 00:41:34.074912 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:34.074867 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27ef8f56-88d3-4b6e-9014-ff92c20bab2f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-72954db1-727gx\" (UID: \"27ef8f56-88d3-4b6e-9014-ff92c20bab2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-72954db1-727gx" Jul 23 00:41:34.075542 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:34.075516 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27ef8f56-88d3-4b6e-9014-ff92c20bab2f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-72954db1-727gx\" (UID: \"27ef8f56-88d3-4b6e-9014-ff92c20bab2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-72954db1-727gx" Jul 23 00:41:34.235826 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:34.235705 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-72954db1-727gx" Jul 23 00:41:34.378066 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:34.378034 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-72954db1-727gx"] Jul 23 00:41:34.380351 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:41:34.380322 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27ef8f56_88d3_4b6e_9014_ff92c20bab2f.slice/crio-c4471eff170ba55d2e846a59ed2cc37f1a501e417bbdc514f6b34aa3a93412a9 WatchSource:0}: Error finding container c4471eff170ba55d2e846a59ed2cc37f1a501e417bbdc514f6b34aa3a93412a9: Status 404 returned error can't find the container with id c4471eff170ba55d2e846a59ed2cc37f1a501e417bbdc514f6b34aa3a93412a9 Jul 23 00:41:35.009583 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:35.009479 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-72954db1-727gx" event={"ID":"27ef8f56-88d3-4b6e-9014-ff92c20bab2f","Type":"ContainerStarted","Data":"6fdb47198cc8e12afdd5d9ed860e793b65886eb07f580a86fd0f9ac30c9c4ba6"} Jul 23 00:41:35.009583 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:35.009550 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-72954db1-727gx" event={"ID":"27ef8f56-88d3-4b6e-9014-ff92c20bab2f","Type":"ContainerStarted","Data":"c4471eff170ba55d2e846a59ed2cc37f1a501e417bbdc514f6b34aa3a93412a9"} Jul 23 00:41:35.025162 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:35.025099 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-72954db1-727gx" podStartSLOduration=2.025080402 podStartE2EDuration="2.025080402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:41:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:41:35.02434197 +0000 UTC m=+2008.819848827" watchObservedRunningTime="2026-07-23 00:41:35.025080402 +0000 UTC m=+2008.820587287" Jul 23 00:41:36.016596 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:36.016558 2562 generic.go:356] "Generic (PLEG): container finished" podID="27ef8f56-88d3-4b6e-9014-ff92c20bab2f" containerID="6fdb47198cc8e12afdd5d9ed860e793b65886eb07f580a86fd0f9ac30c9c4ba6" exitCode=0 Jul 23 00:41:36.017060 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:36.016646 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-72954db1-727gx" event={"ID":"27ef8f56-88d3-4b6e-9014-ff92c20bab2f","Type":"ContainerDied","Data":"6fdb47198cc8e12afdd5d9ed860e793b65886eb07f580a86fd0f9ac30c9c4ba6"} Jul 23 00:41:37.154341 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:37.154314 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-72954db1-727gx" Jul 23 00:41:37.207302 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:37.207253 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27ef8f56-88d3-4b6e-9014-ff92c20bab2f-maas-api-service-ca\") pod \"27ef8f56-88d3-4b6e-9014-ff92c20bab2f\" (UID: \"27ef8f56-88d3-4b6e-9014-ff92c20bab2f\") " Jul 23 00:41:37.207648 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:37.207624 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27ef8f56-88d3-4b6e-9014-ff92c20bab2f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "27ef8f56-88d3-4b6e-9014-ff92c20bab2f" (UID: "27ef8f56-88d3-4b6e-9014-ff92c20bab2f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:41:37.308117 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:37.308019 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27ef8f56-88d3-4b6e-9014-ff92c20bab2f-maas-api-service-ca\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:41:38.029557 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:38.029521 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-72954db1-727gx" Jul 23 00:41:38.029791 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:38.029567 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-72954db1-727gx" event={"ID":"27ef8f56-88d3-4b6e-9014-ff92c20bab2f","Type":"ContainerDied","Data":"c4471eff170ba55d2e846a59ed2cc37f1a501e417bbdc514f6b34aa3a93412a9"} Jul 23 00:41:38.029791 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:38.029618 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4471eff170ba55d2e846a59ed2cc37f1a501e417bbdc514f6b34aa3a93412a9" Jul 23 00:41:48.989819 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:48.989777 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-72954db1-727gx"] Jul 23 00:41:48.996008 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:48.995976 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-72954db1-727gx"] Jul 23 00:41:49.007933 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:49.007904 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27ef8f56-88d3-4b6e-9014-ff92c20bab2f" path="/var/lib/kubelet/pods/27ef8f56-88d3-4b6e-9014-ff92c20bab2f/volumes" Jul 23 00:41:49.971148 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:49.971114 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw"] Jul 23 00:41:49.972100 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:49.972067 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27ef8f56-88d3-4b6e-9014-ff92c20bab2f" containerName="revoke-keys" Jul 23 00:41:49.972100 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:49.972092 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="27ef8f56-88d3-4b6e-9014-ff92c20bab2f" containerName="revoke-keys" Jul 23 00:41:49.972478 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:49.972168 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="daf9c197-91ee-442f-9c32-e257835d26d2" containerName="istio-proxy" Jul 23 00:41:49.972478 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:49.972177 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="27ef8f56-88d3-4b6e-9014-ff92c20bab2f" containerName="revoke-keys" Jul 23 00:41:49.981937 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:49.981894 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw" Jul 23 00:41:49.985100 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:49.985069 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-d0be4a18-openshift-default-dockercfg-ls57v\"" Jul 23 00:41:49.988234 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:49.988203 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw"] Jul 23 00:41:50.139314 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:50.139267 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b294d861-b474-4ee3-a9b2-37faaf34ee9c-istio-data\") pod \"e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw\" (UID: \"b294d861-b474-4ee3-a9b2-37faaf34ee9c\") " pod="openshift-ingress/e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw" Jul 23 00:41:50.139745 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:50.139336 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rprw\" (UniqueName: \"kubernetes.io/projected/b294d861-b474-4ee3-a9b2-37faaf34ee9c-kube-api-access-8rprw\") pod \"e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw\" (UID: \"b294d861-b474-4ee3-a9b2-37faaf34ee9c\") " pod="openshift-ingress/e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw" Jul 23 00:41:50.139745 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:50.139367 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b294d861-b474-4ee3-a9b2-37faaf34ee9c-workload-socket\") pod \"e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw\" (UID: \"b294d861-b474-4ee3-a9b2-37faaf34ee9c\") " pod="openshift-ingress/e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw" Jul 23 00:41:50.139745 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:50.139406 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b294d861-b474-4ee3-a9b2-37faaf34ee9c-istio-podinfo\") pod \"e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw\" (UID: \"b294d861-b474-4ee3-a9b2-37faaf34ee9c\") " pod="openshift-ingress/e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw" Jul 23 00:41:50.139745 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:50.139443 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b294d861-b474-4ee3-a9b2-37faaf34ee9c-workload-certs\") pod \"e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw\" (UID: \"b294d861-b474-4ee3-a9b2-37faaf34ee9c\") " pod="openshift-ingress/e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw" Jul 23 00:41:50.139745 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:50.139501 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b294d861-b474-4ee3-a9b2-37faaf34ee9c-istio-token\") pod \"e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw\" (UID: \"b294d861-b474-4ee3-a9b2-37faaf34ee9c\") " pod="openshift-ingress/e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw" Jul 23 00:41:50.139745 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:50.139553 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b294d861-b474-4ee3-a9b2-37faaf34ee9c-istiod-ca-cert\") pod \"e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw\" (UID: \"b294d861-b474-4ee3-a9b2-37faaf34ee9c\") " pod="openshift-ingress/e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw" Jul 23 00:41:50.139745 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:50.139678 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b294d861-b474-4ee3-a9b2-37faaf34ee9c-credential-socket\") pod \"e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw\" (UID: \"b294d861-b474-4ee3-a9b2-37faaf34ee9c\") " pod="openshift-ingress/e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw" Jul 23 00:41:50.139745 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:50.139735 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b294d861-b474-4ee3-a9b2-37faaf34ee9c-istio-envoy\") pod \"e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw\" (UID: \"b294d861-b474-4ee3-a9b2-37faaf34ee9c\") " pod="openshift-ingress/e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw" Jul 23 00:41:50.241142 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:50.241050 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b294d861-b474-4ee3-a9b2-37faaf34ee9c-istiod-ca-cert\") pod \"e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw\" (UID: \"b294d861-b474-4ee3-a9b2-37faaf34ee9c\") " pod="openshift-ingress/e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw" Jul 23 00:41:50.241142 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:50.241110 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b294d861-b474-4ee3-a9b2-37faaf34ee9c-credential-socket\") pod \"e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw\" (UID: \"b294d861-b474-4ee3-a9b2-37faaf34ee9c\") " pod="openshift-ingress/e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw" Jul 23 00:41:50.241367 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:50.241149 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b294d861-b474-4ee3-a9b2-37faaf34ee9c-istio-envoy\") pod \"e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw\" (UID: \"b294d861-b474-4ee3-a9b2-37faaf34ee9c\") " pod="openshift-ingress/e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw" Jul 23 00:41:50.241367 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:50.241195 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b294d861-b474-4ee3-a9b2-37faaf34ee9c-istio-data\") pod \"e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw\" (UID: \"b294d861-b474-4ee3-a9b2-37faaf34ee9c\") " pod="openshift-ingress/e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw" Jul 23 00:41:50.241367 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:50.241233 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rprw\" (UniqueName: \"kubernetes.io/projected/b294d861-b474-4ee3-a9b2-37faaf34ee9c-kube-api-access-8rprw\") pod \"e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw\" (UID: \"b294d861-b474-4ee3-a9b2-37faaf34ee9c\") " pod="openshift-ingress/e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw" Jul 23 00:41:50.241367 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:50.241259 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b294d861-b474-4ee3-a9b2-37faaf34ee9c-workload-socket\") pod \"e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw\" (UID: \"b294d861-b474-4ee3-a9b2-37faaf34ee9c\") " pod="openshift-ingress/e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw" Jul 23 00:41:50.241367 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:50.241297 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b294d861-b474-4ee3-a9b2-37faaf34ee9c-istio-podinfo\") pod \"e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw\" (UID: \"b294d861-b474-4ee3-a9b2-37faaf34ee9c\") " pod="openshift-ingress/e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw" Jul 23 00:41:50.241367 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:50.241324 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b294d861-b474-4ee3-a9b2-37faaf34ee9c-workload-certs\") pod \"e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw\" (UID: \"b294d861-b474-4ee3-a9b2-37faaf34ee9c\") " pod="openshift-ingress/e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw" Jul 23 00:41:50.241681 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:50.241386 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b294d861-b474-4ee3-a9b2-37faaf34ee9c-istio-token\") pod \"e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw\" (UID: \"b294d861-b474-4ee3-a9b2-37faaf34ee9c\") " pod="openshift-ingress/e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw" Jul 23 00:41:50.241681 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:50.241590 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b294d861-b474-4ee3-a9b2-37faaf34ee9c-credential-socket\") pod \"e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw\" (UID: \"b294d861-b474-4ee3-a9b2-37faaf34ee9c\") " pod="openshift-ingress/e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw" Jul 23 00:41:50.241828 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:50.241690 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b294d861-b474-4ee3-a9b2-37faaf34ee9c-istio-data\") pod \"e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw\" (UID: \"b294d861-b474-4ee3-a9b2-37faaf34ee9c\") " pod="openshift-ingress/e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw" Jul 23 00:41:50.241902 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:50.241874 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b294d861-b474-4ee3-a9b2-37faaf34ee9c-workload-certs\") pod \"e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw\" (UID: \"b294d861-b474-4ee3-a9b2-37faaf34ee9c\") " pod="openshift-ingress/e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw" Jul 23 00:41:50.241971 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:50.241934 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b294d861-b474-4ee3-a9b2-37faaf34ee9c-istiod-ca-cert\") pod \"e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw\" (UID: \"b294d861-b474-4ee3-a9b2-37faaf34ee9c\") " pod="openshift-ingress/e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw" Jul 23 00:41:50.242030 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:50.241994 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b294d861-b474-4ee3-a9b2-37faaf34ee9c-workload-socket\") pod \"e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw\" (UID: \"b294d861-b474-4ee3-a9b2-37faaf34ee9c\") " pod="openshift-ingress/e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw" Jul 23 00:41:50.243620 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:50.243598 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b294d861-b474-4ee3-a9b2-37faaf34ee9c-istio-envoy\") pod \"e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw\" (UID: \"b294d861-b474-4ee3-a9b2-37faaf34ee9c\") " pod="openshift-ingress/e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw" Jul 23 00:41:50.243869 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:50.243851 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b294d861-b474-4ee3-a9b2-37faaf34ee9c-istio-podinfo\") pod \"e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw\" (UID: \"b294d861-b474-4ee3-a9b2-37faaf34ee9c\") " pod="openshift-ingress/e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw" Jul 23 00:41:50.249393 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:50.249372 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rprw\" (UniqueName: \"kubernetes.io/projected/b294d861-b474-4ee3-a9b2-37faaf34ee9c-kube-api-access-8rprw\") pod \"e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw\" (UID: \"b294d861-b474-4ee3-a9b2-37faaf34ee9c\") " pod="openshift-ingress/e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw" Jul 23 00:41:50.249480 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:50.249452 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b294d861-b474-4ee3-a9b2-37faaf34ee9c-istio-token\") pod \"e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw\" (UID: \"b294d861-b474-4ee3-a9b2-37faaf34ee9c\") " pod="openshift-ingress/e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw" Jul 23 00:41:50.295680 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:50.295646 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw" Jul 23 00:41:50.433823 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:50.433794 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw"] Jul 23 00:41:50.435510 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:41:50.435482 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb294d861_b474_4ee3_a9b2_37faaf34ee9c.slice/crio-313176c6a3f1775cb913cab6cc7a5611fa7fcd23386b97c51995f395ae5c9fb3 WatchSource:0}: Error finding container 313176c6a3f1775cb913cab6cc7a5611fa7fcd23386b97c51995f395ae5c9fb3: Status 404 returned error can't find the container with id 313176c6a3f1775cb913cab6cc7a5611fa7fcd23386b97c51995f395ae5c9fb3 Jul 23 00:41:50.437625 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:50.437575 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:41:50.437806 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:50.437640 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:41:50.437806 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:50.437670 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:41:51.104871 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:51.104831 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw" event={"ID":"b294d861-b474-4ee3-a9b2-37faaf34ee9c","Type":"ContainerStarted","Data":"15e422c63447c58169fdbc2b323131c869295782724a4851c361372e212ac9a8"} Jul 23 00:41:51.105059 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:51.104889 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw" event={"ID":"b294d861-b474-4ee3-a9b2-37faaf34ee9c","Type":"ContainerStarted","Data":"313176c6a3f1775cb913cab6cc7a5611fa7fcd23386b97c51995f395ae5c9fb3"} Jul 23 00:41:51.125773 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:51.125699 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw" podStartSLOduration=2.125682239 podStartE2EDuration="2.125682239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:41:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:41:51.12417927 +0000 UTC m=+2024.919686133" watchObservedRunningTime="2026-07-23 00:41:51.125682239 +0000 UTC m=+2024.921189097" Jul 23 00:41:51.296843 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:51.296796 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw" Jul 23 00:41:51.302022 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:51.301994 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw" Jul 23 00:41:52.111209 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:52.111170 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw" Jul 23 00:41:52.112321 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:41:52.112301 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw" Jul 23 00:42:07.354678 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:07.354019 2562 patch_prober.go:28] interesting pod/e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": read tcp 10.134.0.2:42020->10.134.0.42:15021: read: connection reset by peer" start-of-body= Jul 23 00:42:07.354678 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:07.354099 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw" podUID="b294d861-b474-4ee3-a9b2-37faaf34ee9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": read tcp 10.134.0.2:42020->10.134.0.42:15021: read: connection reset by peer" Jul 23 00:42:08.206026 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:08.205992 2562 generic.go:356] "Generic (PLEG): container finished" podID="b294d861-b474-4ee3-a9b2-37faaf34ee9c" containerID="15e422c63447c58169fdbc2b323131c869295782724a4851c361372e212ac9a8" exitCode=0 Jul 23 00:42:08.206288 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:08.206065 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw" event={"ID":"b294d861-b474-4ee3-a9b2-37faaf34ee9c","Type":"ContainerDied","Data":"15e422c63447c58169fdbc2b323131c869295782724a4851c361372e212ac9a8"} Jul 23 00:42:08.206487 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:08.206472 2562 scope.go:117] "RemoveContainer" containerID="15e422c63447c58169fdbc2b323131c869295782724a4851c361372e212ac9a8" Jul 23 00:42:08.207491 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:08.207465 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:42:08.207547 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:08.207533 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:42:08.207589 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:08.207565 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:42:09.214035 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:09.213994 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw" event={"ID":"b294d861-b474-4ee3-a9b2-37faaf34ee9c","Type":"ContainerStarted","Data":"85c4ce5cbabeb88ea6f58df6d93d48ee598cf0d2247bf1433c5a3acf0c7dcae5"} Jul 23 00:42:09.296621 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:09.296574 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw" Jul 23 00:42:09.296895 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:09.296868 2562 patch_prober.go:28] interesting pod/e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 23 00:42:09.296945 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:09.296925 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw" podUID="b294d861-b474-4ee3-a9b2-37faaf34ee9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 23 00:42:10.297067 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:10.297020 2562 patch_prober.go:28] interesting pod/e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 23 00:42:10.297605 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:10.297100 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw" podUID="b294d861-b474-4ee3-a9b2-37faaf34ee9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 23 00:42:11.296389 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:11.296342 2562 patch_prober.go:28] interesting pod/e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 23 00:42:11.296617 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:11.296419 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw" podUID="b294d861-b474-4ee3-a9b2-37faaf34ee9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 23 00:42:12.296579 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:12.296537 2562 patch_prober.go:28] interesting pod/e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 23 00:42:12.298159 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:12.298123 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw" podUID="b294d861-b474-4ee3-a9b2-37faaf34ee9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 23 00:42:13.297058 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:13.297018 2562 patch_prober.go:28] interesting pod/e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 23 00:42:13.297672 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:13.297637 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw" podUID="b294d861-b474-4ee3-a9b2-37faaf34ee9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 23 00:42:14.300084 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:14.300042 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw" Jul 23 00:42:14.300474 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:14.300325 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw" Jul 23 00:42:14.301131 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:14.301115 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw" Jul 23 00:42:34.688943 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:34.688852 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw"] Jul 23 00:42:34.689367 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:34.689198 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw" podUID="b294d861-b474-4ee3-a9b2-37faaf34ee9c" containerName="istio-proxy" containerID="cri-o://85c4ce5cbabeb88ea6f58df6d93d48ee598cf0d2247bf1433c5a3acf0c7dcae5" gracePeriod=30 Jul 23 00:42:35.460473 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:35.460439 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm"] Jul 23 00:42:35.464091 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:35.464071 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm" Jul 23 00:42:35.467075 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:35.467049 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-3fad28dc-openshift-default-dockercfg-6n4ql\"" Jul 23 00:42:35.476870 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:35.476834 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm"] Jul 23 00:42:35.580336 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:35.580289 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56ce11cb-f181-4d9a-ab86-9b6567f73d67-istio-podinfo\") pod \"e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm\" (UID: \"56ce11cb-f181-4d9a-ab86-9b6567f73d67\") " pod="openshift-ingress/e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm" Jul 23 00:42:35.580550 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:35.580345 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6bln\" (UniqueName: \"kubernetes.io/projected/56ce11cb-f181-4d9a-ab86-9b6567f73d67-kube-api-access-k6bln\") pod \"e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm\" (UID: \"56ce11cb-f181-4d9a-ab86-9b6567f73d67\") " pod="openshift-ingress/e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm" Jul 23 00:42:35.580550 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:35.580368 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56ce11cb-f181-4d9a-ab86-9b6567f73d67-istiod-ca-cert\") pod \"e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm\" (UID: \"56ce11cb-f181-4d9a-ab86-9b6567f73d67\") " pod="openshift-ingress/e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm" Jul 23 00:42:35.580550 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:35.580458 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56ce11cb-f181-4d9a-ab86-9b6567f73d67-workload-certs\") pod \"e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm\" (UID: \"56ce11cb-f181-4d9a-ab86-9b6567f73d67\") " pod="openshift-ingress/e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm" Jul 23 00:42:35.580550 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:35.580521 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56ce11cb-f181-4d9a-ab86-9b6567f73d67-istio-data\") pod \"e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm\" (UID: \"56ce11cb-f181-4d9a-ab86-9b6567f73d67\") " pod="openshift-ingress/e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm" Jul 23 00:42:35.580741 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:35.580562 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56ce11cb-f181-4d9a-ab86-9b6567f73d67-istio-envoy\") pod \"e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm\" (UID: \"56ce11cb-f181-4d9a-ab86-9b6567f73d67\") " pod="openshift-ingress/e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm" Jul 23 00:42:35.580741 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:35.580596 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56ce11cb-f181-4d9a-ab86-9b6567f73d67-istio-token\") pod \"e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm\" (UID: \"56ce11cb-f181-4d9a-ab86-9b6567f73d67\") " pod="openshift-ingress/e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm" Jul 23 00:42:35.580741 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:35.580614 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56ce11cb-f181-4d9a-ab86-9b6567f73d67-credential-socket\") pod \"e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm\" (UID: \"56ce11cb-f181-4d9a-ab86-9b6567f73d67\") " pod="openshift-ingress/e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm" Jul 23 00:42:35.580741 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:35.580677 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56ce11cb-f181-4d9a-ab86-9b6567f73d67-workload-socket\") pod \"e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm\" (UID: \"56ce11cb-f181-4d9a-ab86-9b6567f73d67\") " pod="openshift-ingress/e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm" Jul 23 00:42:35.681830 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:35.681741 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56ce11cb-f181-4d9a-ab86-9b6567f73d67-istio-podinfo\") pod \"e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm\" (UID: \"56ce11cb-f181-4d9a-ab86-9b6567f73d67\") " pod="openshift-ingress/e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm" Jul 23 00:42:35.681830 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:35.681843 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6bln\" (UniqueName: \"kubernetes.io/projected/56ce11cb-f181-4d9a-ab86-9b6567f73d67-kube-api-access-k6bln\") pod \"e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm\" (UID: \"56ce11cb-f181-4d9a-ab86-9b6567f73d67\") " pod="openshift-ingress/e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm" Jul 23 00:42:35.682112 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:35.681879 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56ce11cb-f181-4d9a-ab86-9b6567f73d67-istiod-ca-cert\") pod \"e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm\" (UID: \"56ce11cb-f181-4d9a-ab86-9b6567f73d67\") " pod="openshift-ingress/e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm" Jul 23 00:42:35.682112 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:35.681899 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56ce11cb-f181-4d9a-ab86-9b6567f73d67-workload-certs\") pod \"e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm\" (UID: \"56ce11cb-f181-4d9a-ab86-9b6567f73d67\") " pod="openshift-ingress/e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm" Jul 23 00:42:35.682112 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:35.682013 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56ce11cb-f181-4d9a-ab86-9b6567f73d67-istio-data\") pod \"e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm\" (UID: \"56ce11cb-f181-4d9a-ab86-9b6567f73d67\") " pod="openshift-ingress/e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm" Jul 23 00:42:35.682112 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:35.682088 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56ce11cb-f181-4d9a-ab86-9b6567f73d67-istio-envoy\") pod \"e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm\" (UID: \"56ce11cb-f181-4d9a-ab86-9b6567f73d67\") " pod="openshift-ingress/e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm" Jul 23 00:42:35.682336 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:35.682142 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56ce11cb-f181-4d9a-ab86-9b6567f73d67-istio-token\") pod \"e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm\" (UID: \"56ce11cb-f181-4d9a-ab86-9b6567f73d67\") " pod="openshift-ingress/e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm" Jul 23 00:42:35.682336 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:35.682169 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56ce11cb-f181-4d9a-ab86-9b6567f73d67-credential-socket\") pod \"e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm\" (UID: \"56ce11cb-f181-4d9a-ab86-9b6567f73d67\") " pod="openshift-ingress/e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm" Jul 23 00:42:35.682336 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:35.682237 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56ce11cb-f181-4d9a-ab86-9b6567f73d67-workload-socket\") pod \"e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm\" (UID: \"56ce11cb-f181-4d9a-ab86-9b6567f73d67\") " pod="openshift-ingress/e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm" Jul 23 00:42:35.682480 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:35.682376 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56ce11cb-f181-4d9a-ab86-9b6567f73d67-istio-data\") pod \"e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm\" (UID: \"56ce11cb-f181-4d9a-ab86-9b6567f73d67\") " pod="openshift-ingress/e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm" Jul 23 00:42:35.682480 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:35.682435 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56ce11cb-f181-4d9a-ab86-9b6567f73d67-workload-certs\") pod \"e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm\" (UID: \"56ce11cb-f181-4d9a-ab86-9b6567f73d67\") " pod="openshift-ingress/e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm" Jul 23 00:42:35.682656 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:35.682635 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56ce11cb-f181-4d9a-ab86-9b6567f73d67-istiod-ca-cert\") pod \"e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm\" (UID: \"56ce11cb-f181-4d9a-ab86-9b6567f73d67\") " pod="openshift-ingress/e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm" Jul 23 00:42:35.682737 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:35.682713 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56ce11cb-f181-4d9a-ab86-9b6567f73d67-credential-socket\") pod \"e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm\" (UID: \"56ce11cb-f181-4d9a-ab86-9b6567f73d67\") " pod="openshift-ingress/e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm" Jul 23 00:42:35.682828 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:35.682811 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56ce11cb-f181-4d9a-ab86-9b6567f73d67-workload-socket\") pod \"e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm\" (UID: \"56ce11cb-f181-4d9a-ab86-9b6567f73d67\") " pod="openshift-ingress/e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm" Jul 23 00:42:35.684477 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:35.684444 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56ce11cb-f181-4d9a-ab86-9b6567f73d67-istio-podinfo\") pod \"e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm\" (UID: \"56ce11cb-f181-4d9a-ab86-9b6567f73d67\") " pod="openshift-ingress/e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm" Jul 23 00:42:35.684592 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:35.684535 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56ce11cb-f181-4d9a-ab86-9b6567f73d67-istio-envoy\") pod \"e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm\" (UID: \"56ce11cb-f181-4d9a-ab86-9b6567f73d67\") " pod="openshift-ingress/e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm" Jul 23 00:42:35.691035 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:35.691002 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56ce11cb-f181-4d9a-ab86-9b6567f73d67-istio-token\") pod \"e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm\" (UID: \"56ce11cb-f181-4d9a-ab86-9b6567f73d67\") " pod="openshift-ingress/e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm" Jul 23 00:42:35.691374 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:35.691074 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6bln\" (UniqueName: \"kubernetes.io/projected/56ce11cb-f181-4d9a-ab86-9b6567f73d67-kube-api-access-k6bln\") pod \"e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm\" (UID: \"56ce11cb-f181-4d9a-ab86-9b6567f73d67\") " pod="openshift-ingress/e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm" Jul 23 00:42:35.776530 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:35.776493 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm" Jul 23 00:42:35.933197 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:35.933162 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm"] Jul 23 00:42:35.934493 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:42:35.934462 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56ce11cb_f181_4d9a_ab86_9b6567f73d67.slice/crio-9ba2bafeb7c6ad7027b5ead6bc0a23965f5b2bbe238febc4823bc987e5c7df24 WatchSource:0}: Error finding container 9ba2bafeb7c6ad7027b5ead6bc0a23965f5b2bbe238febc4823bc987e5c7df24: Status 404 returned error can't find the container with id 9ba2bafeb7c6ad7027b5ead6bc0a23965f5b2bbe238febc4823bc987e5c7df24 Jul 23 00:42:35.936788 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:35.936736 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:42:35.936885 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:35.936822 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:42:35.936885 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:35.936850 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:42:36.364159 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:36.364121 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm" event={"ID":"56ce11cb-f181-4d9a-ab86-9b6567f73d67","Type":"ContainerStarted","Data":"b8823541b56cef2ed6e540401d6a0dbdfdadf8d108a2f6044cd0387c68a799e1"} Jul 23 00:42:36.364349 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:36.364183 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm" event={"ID":"56ce11cb-f181-4d9a-ab86-9b6567f73d67","Type":"ContainerStarted","Data":"9ba2bafeb7c6ad7027b5ead6bc0a23965f5b2bbe238febc4823bc987e5c7df24"} Jul 23 00:42:36.776831 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:36.776793 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm" Jul 23 00:42:37.781582 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:37.781542 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm" Jul 23 00:42:37.809234 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:37.809178 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm" podStartSLOduration=2.809160912 podStartE2EDuration="2.809160912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:42:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:42:36.382707597 +0000 UTC m=+2070.178214456" watchObservedRunningTime="2026-07-23 00:42:37.809160912 +0000 UTC m=+2071.604667768" Jul 23 00:42:38.376018 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:38.375987 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm" Jul 23 00:42:38.376925 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:38.376904 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm" Jul 23 00:42:39.960111 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:39.960025 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw" Jul 23 00:42:40.127069 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:40.127031 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b294d861-b474-4ee3-a9b2-37faaf34ee9c-workload-certs\") pod \"b294d861-b474-4ee3-a9b2-37faaf34ee9c\" (UID: \"b294d861-b474-4ee3-a9b2-37faaf34ee9c\") " Jul 23 00:42:40.127270 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:40.127093 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b294d861-b474-4ee3-a9b2-37faaf34ee9c-credential-socket\") pod \"b294d861-b474-4ee3-a9b2-37faaf34ee9c\" (UID: \"b294d861-b474-4ee3-a9b2-37faaf34ee9c\") " Jul 23 00:42:40.127270 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:40.127141 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b294d861-b474-4ee3-a9b2-37faaf34ee9c-istiod-ca-cert\") pod \"b294d861-b474-4ee3-a9b2-37faaf34ee9c\" (UID: \"b294d861-b474-4ee3-a9b2-37faaf34ee9c\") " Jul 23 00:42:40.127270 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:40.127163 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b294d861-b474-4ee3-a9b2-37faaf34ee9c-istio-data\") pod \"b294d861-b474-4ee3-a9b2-37faaf34ee9c\" (UID: \"b294d861-b474-4ee3-a9b2-37faaf34ee9c\") " Jul 23 00:42:40.127270 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:40.127198 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b294d861-b474-4ee3-a9b2-37faaf34ee9c-istio-token\") pod \"b294d861-b474-4ee3-a9b2-37faaf34ee9c\" (UID: \"b294d861-b474-4ee3-a9b2-37faaf34ee9c\") " Jul 23 00:42:40.127270 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:40.127246 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b294d861-b474-4ee3-a9b2-37faaf34ee9c-istio-envoy\") pod \"b294d861-b474-4ee3-a9b2-37faaf34ee9c\" (UID: \"b294d861-b474-4ee3-a9b2-37faaf34ee9c\") " Jul 23 00:42:40.127270 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:40.127265 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8rprw\" (UniqueName: \"kubernetes.io/projected/b294d861-b474-4ee3-a9b2-37faaf34ee9c-kube-api-access-8rprw\") pod \"b294d861-b474-4ee3-a9b2-37faaf34ee9c\" (UID: \"b294d861-b474-4ee3-a9b2-37faaf34ee9c\") " Jul 23 00:42:40.127612 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:40.127307 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b294d861-b474-4ee3-a9b2-37faaf34ee9c-workload-socket\") pod \"b294d861-b474-4ee3-a9b2-37faaf34ee9c\" (UID: \"b294d861-b474-4ee3-a9b2-37faaf34ee9c\") " Jul 23 00:42:40.127612 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:40.127352 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b294d861-b474-4ee3-a9b2-37faaf34ee9c-istio-podinfo\") pod \"b294d861-b474-4ee3-a9b2-37faaf34ee9c\" (UID: \"b294d861-b474-4ee3-a9b2-37faaf34ee9c\") " Jul 23 00:42:40.127612 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:40.127384 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b294d861-b474-4ee3-a9b2-37faaf34ee9c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b294d861-b474-4ee3-a9b2-37faaf34ee9c" (UID: "b294d861-b474-4ee3-a9b2-37faaf34ee9c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:42:40.127612 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:40.127501 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b294d861-b474-4ee3-a9b2-37faaf34ee9c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b294d861-b474-4ee3-a9b2-37faaf34ee9c" (UID: "b294d861-b474-4ee3-a9b2-37faaf34ee9c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:42:40.127849 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:40.127640 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b294d861-b474-4ee3-a9b2-37faaf34ee9c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b294d861-b474-4ee3-a9b2-37faaf34ee9c" (UID: "b294d861-b474-4ee3-a9b2-37faaf34ee9c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:42:40.127849 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:40.127650 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b294d861-b474-4ee3-a9b2-37faaf34ee9c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b294d861-b474-4ee3-a9b2-37faaf34ee9c" (UID: "b294d861-b474-4ee3-a9b2-37faaf34ee9c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:42:40.128062 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:40.128025 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b294d861-b474-4ee3-a9b2-37faaf34ee9c-workload-certs\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:42:40.128062 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:40.128051 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b294d861-b474-4ee3-a9b2-37faaf34ee9c-credential-socket\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:42:40.128170 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:40.128067 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b294d861-b474-4ee3-a9b2-37faaf34ee9c-istiod-ca-cert\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:42:40.128170 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:40.128081 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b294d861-b474-4ee3-a9b2-37faaf34ee9c-workload-socket\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:42:40.128281 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:40.128247 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b294d861-b474-4ee3-a9b2-37faaf34ee9c-istio-data" (OuterVolumeSpecName: "istio-data") pod "b294d861-b474-4ee3-a9b2-37faaf34ee9c" (UID: "b294d861-b474-4ee3-a9b2-37faaf34ee9c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:42:40.129869 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:40.129835 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b294d861-b474-4ee3-a9b2-37faaf34ee9c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b294d861-b474-4ee3-a9b2-37faaf34ee9c" (UID: "b294d861-b474-4ee3-a9b2-37faaf34ee9c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:42:40.129990 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:40.129921 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b294d861-b474-4ee3-a9b2-37faaf34ee9c-kube-api-access-8rprw" (OuterVolumeSpecName: "kube-api-access-8rprw") pod "b294d861-b474-4ee3-a9b2-37faaf34ee9c" (UID: "b294d861-b474-4ee3-a9b2-37faaf34ee9c"). InnerVolumeSpecName "kube-api-access-8rprw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:42:40.129990 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:40.129958 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b294d861-b474-4ee3-a9b2-37faaf34ee9c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b294d861-b474-4ee3-a9b2-37faaf34ee9c" (UID: "b294d861-b474-4ee3-a9b2-37faaf34ee9c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 00:42:40.130080 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:40.130063 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b294d861-b474-4ee3-a9b2-37faaf34ee9c-istio-token" (OuterVolumeSpecName: "istio-token") pod "b294d861-b474-4ee3-a9b2-37faaf34ee9c" (UID: "b294d861-b474-4ee3-a9b2-37faaf34ee9c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:42:40.229471 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:40.229366 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b294d861-b474-4ee3-a9b2-37faaf34ee9c-istio-podinfo\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:42:40.229471 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:40.229403 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b294d861-b474-4ee3-a9b2-37faaf34ee9c-istio-data\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:42:40.229471 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:40.229412 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b294d861-b474-4ee3-a9b2-37faaf34ee9c-istio-token\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:42:40.229471 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:40.229420 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b294d861-b474-4ee3-a9b2-37faaf34ee9c-istio-envoy\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:42:40.229471 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:40.229429 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rprw\" (UniqueName: \"kubernetes.io/projected/b294d861-b474-4ee3-a9b2-37faaf34ee9c-kube-api-access-8rprw\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:42:40.389835 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:40.389799 2562 generic.go:356] "Generic (PLEG): container finished" podID="b294d861-b474-4ee3-a9b2-37faaf34ee9c" containerID="85c4ce5cbabeb88ea6f58df6d93d48ee598cf0d2247bf1433c5a3acf0c7dcae5" exitCode=0 Jul 23 00:42:40.390024 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:40.389850 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw" Jul 23 00:42:40.390024 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:40.389896 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw" event={"ID":"b294d861-b474-4ee3-a9b2-37faaf34ee9c","Type":"ContainerDied","Data":"85c4ce5cbabeb88ea6f58df6d93d48ee598cf0d2247bf1433c5a3acf0c7dcae5"} Jul 23 00:42:40.390024 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:40.389959 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw" event={"ID":"b294d861-b474-4ee3-a9b2-37faaf34ee9c","Type":"ContainerDied","Data":"313176c6a3f1775cb913cab6cc7a5611fa7fcd23386b97c51995f395ae5c9fb3"} Jul 23 00:42:40.390024 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:40.389983 2562 scope.go:117] "RemoveContainer" containerID="85c4ce5cbabeb88ea6f58df6d93d48ee598cf0d2247bf1433c5a3acf0c7dcae5" Jul 23 00:42:40.400438 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:40.400414 2562 scope.go:117] "RemoveContainer" containerID="15e422c63447c58169fdbc2b323131c869295782724a4851c361372e212ac9a8" Jul 23 00:42:40.409408 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:40.409379 2562 scope.go:117] "RemoveContainer" containerID="85c4ce5cbabeb88ea6f58df6d93d48ee598cf0d2247bf1433c5a3acf0c7dcae5" Jul 23 00:42:40.409745 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:42:40.409724 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85c4ce5cbabeb88ea6f58df6d93d48ee598cf0d2247bf1433c5a3acf0c7dcae5\": container with ID starting with 85c4ce5cbabeb88ea6f58df6d93d48ee598cf0d2247bf1433c5a3acf0c7dcae5 not found: ID does not exist" containerID="85c4ce5cbabeb88ea6f58df6d93d48ee598cf0d2247bf1433c5a3acf0c7dcae5" Jul 23 00:42:40.409824 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:40.409773 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85c4ce5cbabeb88ea6f58df6d93d48ee598cf0d2247bf1433c5a3acf0c7dcae5"} err="failed to get container status \"85c4ce5cbabeb88ea6f58df6d93d48ee598cf0d2247bf1433c5a3acf0c7dcae5\": rpc error: code = NotFound desc = could not find container \"85c4ce5cbabeb88ea6f58df6d93d48ee598cf0d2247bf1433c5a3acf0c7dcae5\": container with ID starting with 85c4ce5cbabeb88ea6f58df6d93d48ee598cf0d2247bf1433c5a3acf0c7dcae5 not found: ID does not exist" Jul 23 00:42:40.409824 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:40.409795 2562 scope.go:117] "RemoveContainer" containerID="15e422c63447c58169fdbc2b323131c869295782724a4851c361372e212ac9a8" Jul 23 00:42:40.410070 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:42:40.410048 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15e422c63447c58169fdbc2b323131c869295782724a4851c361372e212ac9a8\": container with ID starting with 15e422c63447c58169fdbc2b323131c869295782724a4851c361372e212ac9a8 not found: ID does not exist" containerID="15e422c63447c58169fdbc2b323131c869295782724a4851c361372e212ac9a8" Jul 23 00:42:40.410140 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:40.410079 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15e422c63447c58169fdbc2b323131c869295782724a4851c361372e212ac9a8"} err="failed to get container status \"15e422c63447c58169fdbc2b323131c869295782724a4851c361372e212ac9a8\": rpc error: code = NotFound desc = could not find container \"15e422c63447c58169fdbc2b323131c869295782724a4851c361372e212ac9a8\": container with ID starting with 15e422c63447c58169fdbc2b323131c869295782724a4851c361372e212ac9a8 not found: ID does not exist" Jul 23 00:42:40.415118 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:40.415082 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw"] Jul 23 00:42:40.417861 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:40.417832 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-d0be4a18-openshift-default-69bdc9c87c-gkwzw"] Jul 23 00:42:41.007918 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:41.007878 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b294d861-b474-4ee3-a9b2-37faaf34ee9c" path="/var/lib/kubelet/pods/b294d861-b474-4ee3-a9b2-37faaf34ee9c/volumes" Jul 23 00:42:52.812051 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:42:52.812010 2562 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56ce11cb_f181_4d9a_ab86_9b6567f73d67.slice/crio-conmon-b8823541b56cef2ed6e540401d6a0dbdfdadf8d108a2f6044cd0387c68a799e1.scope\": RecentStats: unable to find data in memory cache], [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@2-23008-0.service\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56ce11cb_f181_4d9a_ab86_9b6567f73d67.slice/crio-b8823541b56cef2ed6e540401d6a0dbdfdadf8d108a2f6044cd0387c68a799e1.scope\": RecentStats: unable to find data in memory cache]" Jul 23 00:42:53.377223 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:53.377184 2562 patch_prober.go:28] interesting pod/e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 23 00:42:53.377411 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:53.377245 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm" podUID="56ce11cb-f181-4d9a-ab86-9b6567f73d67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 23 00:42:53.469337 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:53.469303 2562 generic.go:356] "Generic (PLEG): container finished" podID="56ce11cb-f181-4d9a-ab86-9b6567f73d67" containerID="b8823541b56cef2ed6e540401d6a0dbdfdadf8d108a2f6044cd0387c68a799e1" exitCode=0 Jul 23 00:42:53.469497 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:53.469376 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm" event={"ID":"56ce11cb-f181-4d9a-ab86-9b6567f73d67","Type":"ContainerDied","Data":"b8823541b56cef2ed6e540401d6a0dbdfdadf8d108a2f6044cd0387c68a799e1"} Jul 23 00:42:53.469885 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:53.469863 2562 scope.go:117] "RemoveContainer" containerID="b8823541b56cef2ed6e540401d6a0dbdfdadf8d108a2f6044cd0387c68a799e1" Jul 23 00:42:53.470979 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:53.470949 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:42:53.471096 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:53.471040 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:42:53.471160 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:53.471091 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:42:54.477379 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:54.477340 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm" event={"ID":"56ce11cb-f181-4d9a-ab86-9b6567f73d67","Type":"ContainerStarted","Data":"6cc73e6d33240d5429b9b70feb796bb65e56b22ee7553b232ca3755cca09eaef"} Jul 23 00:42:54.777174 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:54.777134 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm" Jul 23 00:42:54.777457 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:54.777430 2562 patch_prober.go:28] interesting pod/e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 23 00:42:54.777567 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:54.777489 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm" podUID="56ce11cb-f181-4d9a-ab86-9b6567f73d67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 23 00:42:55.777850 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:55.777810 2562 patch_prober.go:28] interesting pod/e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 23 00:42:55.778244 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:55.777869 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm" podUID="56ce11cb-f181-4d9a-ab86-9b6567f73d67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 23 00:42:56.777963 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:56.777914 2562 patch_prober.go:28] interesting pod/e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 23 00:42:56.778456 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:56.777996 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm" podUID="56ce11cb-f181-4d9a-ab86-9b6567f73d67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 23 00:42:57.777112 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:57.777076 2562 patch_prober.go:28] interesting pod/e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 23 00:42:57.777308 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:57.777147 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm" podUID="56ce11cb-f181-4d9a-ab86-9b6567f73d67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 23 00:42:58.781411 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:58.781379 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm" Jul 23 00:42:58.781864 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:58.781783 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm" Jul 23 00:42:58.782555 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:42:58.782538 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm" Jul 23 00:43:17.867803 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:43:17.867743 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm"] Jul 23 00:43:17.868872 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:43:17.868809 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm" podUID="56ce11cb-f181-4d9a-ab86-9b6567f73d67" containerName="istio-proxy" containerID="cri-o://6cc73e6d33240d5429b9b70feb796bb65e56b22ee7553b232ca3755cca09eaef" gracePeriod=30 Jul 23 00:43:22.922263 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:43:22.922225 2562 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56ce11cb_f181_4d9a_ab86_9b6567f73d67.slice/crio-conmon-6cc73e6d33240d5429b9b70feb796bb65e56b22ee7553b232ca3755cca09eaef.scope\": RecentStats: unable to find data in memory cache]" Jul 23 00:43:23.126072 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:43:23.126042 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm" Jul 23 00:43:23.163001 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:43:23.162901 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56ce11cb-f181-4d9a-ab86-9b6567f73d67-istio-envoy\") pod \"56ce11cb-f181-4d9a-ab86-9b6567f73d67\" (UID: \"56ce11cb-f181-4d9a-ab86-9b6567f73d67\") " Jul 23 00:43:23.163001 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:43:23.162951 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56ce11cb-f181-4d9a-ab86-9b6567f73d67-workload-certs\") pod \"56ce11cb-f181-4d9a-ab86-9b6567f73d67\" (UID: \"56ce11cb-f181-4d9a-ab86-9b6567f73d67\") " Jul 23 00:43:23.163001 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:43:23.162973 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56ce11cb-f181-4d9a-ab86-9b6567f73d67-workload-socket\") pod \"56ce11cb-f181-4d9a-ab86-9b6567f73d67\" (UID: \"56ce11cb-f181-4d9a-ab86-9b6567f73d67\") " Jul 23 00:43:23.163001 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:43:23.162998 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56ce11cb-f181-4d9a-ab86-9b6567f73d67-istio-podinfo\") pod \"56ce11cb-f181-4d9a-ab86-9b6567f73d67\" (UID: \"56ce11cb-f181-4d9a-ab86-9b6567f73d67\") " Jul 23 00:43:23.163354 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:43:23.163020 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56ce11cb-f181-4d9a-ab86-9b6567f73d67-istiod-ca-cert\") pod \"56ce11cb-f181-4d9a-ab86-9b6567f73d67\" (UID: \"56ce11cb-f181-4d9a-ab86-9b6567f73d67\") " Jul 23 00:43:23.163354 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:43:23.163081 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k6bln\" (UniqueName: \"kubernetes.io/projected/56ce11cb-f181-4d9a-ab86-9b6567f73d67-kube-api-access-k6bln\") pod \"56ce11cb-f181-4d9a-ab86-9b6567f73d67\" (UID: \"56ce11cb-f181-4d9a-ab86-9b6567f73d67\") " Jul 23 00:43:23.163354 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:43:23.163108 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56ce11cb-f181-4d9a-ab86-9b6567f73d67-credential-socket\") pod \"56ce11cb-f181-4d9a-ab86-9b6567f73d67\" (UID: \"56ce11cb-f181-4d9a-ab86-9b6567f73d67\") " Jul 23 00:43:23.163354 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:43:23.163271 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56ce11cb-f181-4d9a-ab86-9b6567f73d67-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "56ce11cb-f181-4d9a-ab86-9b6567f73d67" (UID: "56ce11cb-f181-4d9a-ab86-9b6567f73d67"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:43:23.163574 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:43:23.163360 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56ce11cb-f181-4d9a-ab86-9b6567f73d67-istio-token\") pod \"56ce11cb-f181-4d9a-ab86-9b6567f73d67\" (UID: \"56ce11cb-f181-4d9a-ab86-9b6567f73d67\") " Jul 23 00:43:23.163574 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:43:23.163406 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56ce11cb-f181-4d9a-ab86-9b6567f73d67-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "56ce11cb-f181-4d9a-ab86-9b6567f73d67" (UID: "56ce11cb-f181-4d9a-ab86-9b6567f73d67"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:43:23.163574 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:43:23.163427 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56ce11cb-f181-4d9a-ab86-9b6567f73d67-istio-data\") pod \"56ce11cb-f181-4d9a-ab86-9b6567f73d67\" (UID: \"56ce11cb-f181-4d9a-ab86-9b6567f73d67\") " Jul 23 00:43:23.163574 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:43:23.163433 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56ce11cb-f181-4d9a-ab86-9b6567f73d67-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "56ce11cb-f181-4d9a-ab86-9b6567f73d67" (UID: "56ce11cb-f181-4d9a-ab86-9b6567f73d67"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:43:23.163876 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:43:23.163838 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56ce11cb-f181-4d9a-ab86-9b6567f73d67-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "56ce11cb-f181-4d9a-ab86-9b6567f73d67" (UID: "56ce11cb-f181-4d9a-ab86-9b6567f73d67"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:43:23.164017 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:43:23.163903 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56ce11cb-f181-4d9a-ab86-9b6567f73d67-workload-certs\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:43:23.164017 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:43:23.163927 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56ce11cb-f181-4d9a-ab86-9b6567f73d67-workload-socket\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:43:23.164017 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:43:23.163942 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56ce11cb-f181-4d9a-ab86-9b6567f73d67-credential-socket\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:43:23.164533 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:43:23.164504 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56ce11cb-f181-4d9a-ab86-9b6567f73d67-istio-data" (OuterVolumeSpecName: "istio-data") pod "56ce11cb-f181-4d9a-ab86-9b6567f73d67" (UID: "56ce11cb-f181-4d9a-ab86-9b6567f73d67"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:43:23.165553 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:43:23.165521 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/56ce11cb-f181-4d9a-ab86-9b6567f73d67-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "56ce11cb-f181-4d9a-ab86-9b6567f73d67" (UID: "56ce11cb-f181-4d9a-ab86-9b6567f73d67"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 00:43:23.165679 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:43:23.165626 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56ce11cb-f181-4d9a-ab86-9b6567f73d67-istio-token" (OuterVolumeSpecName: "istio-token") pod "56ce11cb-f181-4d9a-ab86-9b6567f73d67" (UID: "56ce11cb-f181-4d9a-ab86-9b6567f73d67"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:43:23.165740 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:43:23.165694 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56ce11cb-f181-4d9a-ab86-9b6567f73d67-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "56ce11cb-f181-4d9a-ab86-9b6567f73d67" (UID: "56ce11cb-f181-4d9a-ab86-9b6567f73d67"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:43:23.165740 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:43:23.165727 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56ce11cb-f181-4d9a-ab86-9b6567f73d67-kube-api-access-k6bln" (OuterVolumeSpecName: "kube-api-access-k6bln") pod "56ce11cb-f181-4d9a-ab86-9b6567f73d67" (UID: "56ce11cb-f181-4d9a-ab86-9b6567f73d67"). InnerVolumeSpecName "kube-api-access-k6bln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:43:23.264610 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:43:23.264568 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56ce11cb-f181-4d9a-ab86-9b6567f73d67-istio-envoy\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:43:23.264610 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:43:23.264603 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56ce11cb-f181-4d9a-ab86-9b6567f73d67-istio-podinfo\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:43:23.264610 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:43:23.264614 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56ce11cb-f181-4d9a-ab86-9b6567f73d67-istiod-ca-cert\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:43:23.264610 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:43:23.264622 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6bln\" (UniqueName: \"kubernetes.io/projected/56ce11cb-f181-4d9a-ab86-9b6567f73d67-kube-api-access-k6bln\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:43:23.264939 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:43:23.264631 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56ce11cb-f181-4d9a-ab86-9b6567f73d67-istio-token\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:43:23.264939 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:43:23.264640 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56ce11cb-f181-4d9a-ab86-9b6567f73d67-istio-data\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:43:23.641489 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:43:23.641441 2562 generic.go:356] "Generic (PLEG): container finished" podID="56ce11cb-f181-4d9a-ab86-9b6567f73d67" containerID="6cc73e6d33240d5429b9b70feb796bb65e56b22ee7553b232ca3755cca09eaef" exitCode=0 Jul 23 00:43:23.641700 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:43:23.641497 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm" event={"ID":"56ce11cb-f181-4d9a-ab86-9b6567f73d67","Type":"ContainerDied","Data":"6cc73e6d33240d5429b9b70feb796bb65e56b22ee7553b232ca3755cca09eaef"} Jul 23 00:43:23.641700 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:43:23.641507 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm" Jul 23 00:43:23.641700 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:43:23.641582 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm" event={"ID":"56ce11cb-f181-4d9a-ab86-9b6567f73d67","Type":"ContainerDied","Data":"9ba2bafeb7c6ad7027b5ead6bc0a23965f5b2bbe238febc4823bc987e5c7df24"} Jul 23 00:43:23.641700 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:43:23.641612 2562 scope.go:117] "RemoveContainer" containerID="6cc73e6d33240d5429b9b70feb796bb65e56b22ee7553b232ca3755cca09eaef" Jul 23 00:43:23.651630 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:43:23.651607 2562 scope.go:117] "RemoveContainer" containerID="b8823541b56cef2ed6e540401d6a0dbdfdadf8d108a2f6044cd0387c68a799e1" Jul 23 00:43:23.661933 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:43:23.661906 2562 scope.go:117] "RemoveContainer" containerID="6cc73e6d33240d5429b9b70feb796bb65e56b22ee7553b232ca3755cca09eaef" Jul 23 00:43:23.662258 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:43:23.662236 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cc73e6d33240d5429b9b70feb796bb65e56b22ee7553b232ca3755cca09eaef\": container with ID starting with 6cc73e6d33240d5429b9b70feb796bb65e56b22ee7553b232ca3755cca09eaef not found: ID does not exist" containerID="6cc73e6d33240d5429b9b70feb796bb65e56b22ee7553b232ca3755cca09eaef" Jul 23 00:43:23.662322 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:43:23.662268 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cc73e6d33240d5429b9b70feb796bb65e56b22ee7553b232ca3755cca09eaef"} err="failed to get container status \"6cc73e6d33240d5429b9b70feb796bb65e56b22ee7553b232ca3755cca09eaef\": rpc error: code = NotFound desc = could not find container \"6cc73e6d33240d5429b9b70feb796bb65e56b22ee7553b232ca3755cca09eaef\": container with ID starting with 6cc73e6d33240d5429b9b70feb796bb65e56b22ee7553b232ca3755cca09eaef not found: ID does not exist" Jul 23 00:43:23.662322 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:43:23.662288 2562 scope.go:117] "RemoveContainer" containerID="b8823541b56cef2ed6e540401d6a0dbdfdadf8d108a2f6044cd0387c68a799e1" Jul 23 00:43:23.662586 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:43:23.662564 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8823541b56cef2ed6e540401d6a0dbdfdadf8d108a2f6044cd0387c68a799e1\": container with ID starting with b8823541b56cef2ed6e540401d6a0dbdfdadf8d108a2f6044cd0387c68a799e1 not found: ID does not exist" containerID="b8823541b56cef2ed6e540401d6a0dbdfdadf8d108a2f6044cd0387c68a799e1" Jul 23 00:43:23.662633 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:43:23.662593 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8823541b56cef2ed6e540401d6a0dbdfdadf8d108a2f6044cd0387c68a799e1"} err="failed to get container status \"b8823541b56cef2ed6e540401d6a0dbdfdadf8d108a2f6044cd0387c68a799e1\": rpc error: code = NotFound desc = could not find container \"b8823541b56cef2ed6e540401d6a0dbdfdadf8d108a2f6044cd0387c68a799e1\": container with ID starting with b8823541b56cef2ed6e540401d6a0dbdfdadf8d108a2f6044cd0387c68a799e1 not found: ID does not exist" Jul 23 00:43:23.666500 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:43:23.666467 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm"] Jul 23 00:43:23.670456 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:43:23.670426 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-3fad28dc-openshift-default-787d89bf79-p4tqm"] Jul 23 00:43:25.007386 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:43:25.007350 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56ce11cb-f181-4d9a-ab86-9b6567f73d67" path="/var/lib/kubelet/pods/56ce11cb-f181-4d9a-ab86-9b6567f73d67/volumes" Jul 23 00:45:00.139680 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:00.139631 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29746125-s4rhn"] Jul 23 00:45:00.140344 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:00.140228 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b294d861-b474-4ee3-a9b2-37faaf34ee9c" containerName="istio-proxy" Jul 23 00:45:00.140344 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:00.140253 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b294d861-b474-4ee3-a9b2-37faaf34ee9c" containerName="istio-proxy" Jul 23 00:45:00.140344 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:00.140294 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56ce11cb-f181-4d9a-ab86-9b6567f73d67" containerName="istio-proxy" Jul 23 00:45:00.140344 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:00.140304 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="56ce11cb-f181-4d9a-ab86-9b6567f73d67" containerName="istio-proxy" Jul 23 00:45:00.140344 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:00.140327 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56ce11cb-f181-4d9a-ab86-9b6567f73d67" containerName="istio-proxy" Jul 23 00:45:00.140344 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:00.140335 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="56ce11cb-f181-4d9a-ab86-9b6567f73d67" containerName="istio-proxy" Jul 23 00:45:00.140657 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:00.140420 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b294d861-b474-4ee3-a9b2-37faaf34ee9c" containerName="istio-proxy" Jul 23 00:45:00.140657 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:00.140433 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="56ce11cb-f181-4d9a-ab86-9b6567f73d67" containerName="istio-proxy" Jul 23 00:45:00.140657 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:00.140447 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="56ce11cb-f181-4d9a-ab86-9b6567f73d67" containerName="istio-proxy" Jul 23 00:45:00.140657 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:00.140458 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b294d861-b474-4ee3-a9b2-37faaf34ee9c" containerName="istio-proxy" Jul 23 00:45:00.143521 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:00.143496 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746125-s4rhn" Jul 23 00:45:00.146209 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:00.146182 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w8tk2\"" Jul 23 00:45:00.153526 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:00.153502 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746125-s4rhn" Jul 23 00:45:00.168831 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:00.167283 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29746125-s4rhn"] Jul 23 00:45:00.309149 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:00.309116 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29746125-s4rhn"] Jul 23 00:45:00.311656 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:45:00.311627 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60a410ac_a77e_47e3_8b3c_31bceacecc1d.slice/crio-f7dbfc578aa489d40d7a92cebc6e81c7f74af6e11b4bab1c47820e05e41ad458 WatchSource:0}: Error finding container f7dbfc578aa489d40d7a92cebc6e81c7f74af6e11b4bab1c47820e05e41ad458: Status 404 returned error can't find the container with id f7dbfc578aa489d40d7a92cebc6e81c7f74af6e11b4bab1c47820e05e41ad458 Jul 23 00:45:01.188725 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:01.188682 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746125-s4rhn" event={"ID":"60a410ac-a77e-47e3-8b3c-31bceacecc1d","Type":"ContainerStarted","Data":"c1b3fc1b77c853f5cce4786c9d4d41cf2ca46abdb4d0738b7d0adf96449ff02c"} Jul 23 00:45:01.189164 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:01.188749 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746125-s4rhn" event={"ID":"60a410ac-a77e-47e3-8b3c-31bceacecc1d","Type":"ContainerStarted","Data":"f7dbfc578aa489d40d7a92cebc6e81c7f74af6e11b4bab1c47820e05e41ad458"} Jul 23 00:45:01.208216 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:01.208158 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746125-s4rhn" podStartSLOduration=1.208141379 podStartE2EDuration="1.208141379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:45:01.206216064 +0000 UTC m=+2215.001722923" watchObservedRunningTime="2026-07-23 00:45:01.208141379 +0000 UTC m=+2215.003648236" Jul 23 00:45:01.249069 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:01.249029 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm"] Jul 23 00:45:01.249665 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:01.249647 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b294d861-b474-4ee3-a9b2-37faaf34ee9c" containerName="istio-proxy" Jul 23 00:45:01.249781 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:01.249668 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b294d861-b474-4ee3-a9b2-37faaf34ee9c" containerName="istio-proxy" Jul 23 00:45:01.253342 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:01.253319 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm" Jul 23 00:45:01.256108 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:01.256080 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-c65690b4-openshift-default-dockercfg-286wx\"" Jul 23 00:45:01.263818 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:01.263785 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm"] Jul 23 00:45:01.384726 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:01.384678 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c34a7a9-02a7-4d3b-b10c-610b83df5f3f-istio-data\") pod \"e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm\" (UID: \"8c34a7a9-02a7-4d3b-b10c-610b83df5f3f\") " pod="openshift-ingress/e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm" Jul 23 00:45:01.384929 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:01.384749 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c34a7a9-02a7-4d3b-b10c-610b83df5f3f-workload-certs\") pod \"e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm\" (UID: \"8c34a7a9-02a7-4d3b-b10c-610b83df5f3f\") " pod="openshift-ingress/e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm" Jul 23 00:45:01.384929 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:01.384845 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c34a7a9-02a7-4d3b-b10c-610b83df5f3f-workload-socket\") pod \"e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm\" (UID: \"8c34a7a9-02a7-4d3b-b10c-610b83df5f3f\") " pod="openshift-ingress/e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm" Jul 23 00:45:01.384929 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:01.384902 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c34a7a9-02a7-4d3b-b10c-610b83df5f3f-credential-socket\") pod \"e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm\" (UID: \"8c34a7a9-02a7-4d3b-b10c-610b83df5f3f\") " pod="openshift-ingress/e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm" Jul 23 00:45:01.384929 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:01.384924 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkfmz\" (UniqueName: \"kubernetes.io/projected/8c34a7a9-02a7-4d3b-b10c-610b83df5f3f-kube-api-access-hkfmz\") pod \"e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm\" (UID: \"8c34a7a9-02a7-4d3b-b10c-610b83df5f3f\") " pod="openshift-ingress/e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm" Jul 23 00:45:01.385098 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:01.384946 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c34a7a9-02a7-4d3b-b10c-610b83df5f3f-istiod-ca-cert\") pod \"e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm\" (UID: \"8c34a7a9-02a7-4d3b-b10c-610b83df5f3f\") " pod="openshift-ingress/e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm" Jul 23 00:45:01.385098 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:01.385016 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c34a7a9-02a7-4d3b-b10c-610b83df5f3f-istio-token\") pod \"e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm\" (UID: \"8c34a7a9-02a7-4d3b-b10c-610b83df5f3f\") " pod="openshift-ingress/e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm" Jul 23 00:45:01.385098 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:01.385069 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c34a7a9-02a7-4d3b-b10c-610b83df5f3f-istio-podinfo\") pod \"e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm\" (UID: \"8c34a7a9-02a7-4d3b-b10c-610b83df5f3f\") " pod="openshift-ingress/e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm" Jul 23 00:45:01.385202 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:01.385103 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c34a7a9-02a7-4d3b-b10c-610b83df5f3f-istio-envoy\") pod \"e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm\" (UID: \"8c34a7a9-02a7-4d3b-b10c-610b83df5f3f\") " pod="openshift-ingress/e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm" Jul 23 00:45:01.486609 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:01.486517 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c34a7a9-02a7-4d3b-b10c-610b83df5f3f-workload-socket\") pod \"e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm\" (UID: \"8c34a7a9-02a7-4d3b-b10c-610b83df5f3f\") " pod="openshift-ingress/e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm" Jul 23 00:45:01.486609 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:01.486562 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c34a7a9-02a7-4d3b-b10c-610b83df5f3f-credential-socket\") pod \"e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm\" (UID: \"8c34a7a9-02a7-4d3b-b10c-610b83df5f3f\") " pod="openshift-ingress/e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm" Jul 23 00:45:01.486890 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:01.486700 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkfmz\" (UniqueName: \"kubernetes.io/projected/8c34a7a9-02a7-4d3b-b10c-610b83df5f3f-kube-api-access-hkfmz\") pod \"e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm\" (UID: \"8c34a7a9-02a7-4d3b-b10c-610b83df5f3f\") " pod="openshift-ingress/e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm" Jul 23 00:45:01.486890 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:01.486793 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c34a7a9-02a7-4d3b-b10c-610b83df5f3f-istiod-ca-cert\") pod \"e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm\" (UID: \"8c34a7a9-02a7-4d3b-b10c-610b83df5f3f\") " pod="openshift-ingress/e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm" Jul 23 00:45:01.486890 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:01.486856 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c34a7a9-02a7-4d3b-b10c-610b83df5f3f-istio-token\") pod \"e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm\" (UID: \"8c34a7a9-02a7-4d3b-b10c-610b83df5f3f\") " pod="openshift-ingress/e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm" Jul 23 00:45:01.487060 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:01.486897 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c34a7a9-02a7-4d3b-b10c-610b83df5f3f-istio-podinfo\") pod \"e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm\" (UID: \"8c34a7a9-02a7-4d3b-b10c-610b83df5f3f\") " pod="openshift-ingress/e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm" Jul 23 00:45:01.487060 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:01.486956 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c34a7a9-02a7-4d3b-b10c-610b83df5f3f-workload-socket\") pod \"e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm\" (UID: \"8c34a7a9-02a7-4d3b-b10c-610b83df5f3f\") " pod="openshift-ingress/e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm" Jul 23 00:45:01.487060 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:01.486959 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c34a7a9-02a7-4d3b-b10c-610b83df5f3f-istio-envoy\") pod \"e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm\" (UID: \"8c34a7a9-02a7-4d3b-b10c-610b83df5f3f\") " pod="openshift-ingress/e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm" Jul 23 00:45:01.487060 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:01.486994 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c34a7a9-02a7-4d3b-b10c-610b83df5f3f-credential-socket\") pod \"e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm\" (UID: \"8c34a7a9-02a7-4d3b-b10c-610b83df5f3f\") " pod="openshift-ingress/e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm" Jul 23 00:45:01.487060 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:01.487064 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c34a7a9-02a7-4d3b-b10c-610b83df5f3f-istio-data\") pod \"e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm\" (UID: \"8c34a7a9-02a7-4d3b-b10c-610b83df5f3f\") " pod="openshift-ingress/e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm" Jul 23 00:45:01.487349 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:01.487117 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c34a7a9-02a7-4d3b-b10c-610b83df5f3f-workload-certs\") pod \"e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm\" (UID: \"8c34a7a9-02a7-4d3b-b10c-610b83df5f3f\") " pod="openshift-ingress/e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm" Jul 23 00:45:01.487349 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:01.487333 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c34a7a9-02a7-4d3b-b10c-610b83df5f3f-istio-data\") pod \"e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm\" (UID: \"8c34a7a9-02a7-4d3b-b10c-610b83df5f3f\") " pod="openshift-ingress/e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm" Jul 23 00:45:01.487464 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:01.487404 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c34a7a9-02a7-4d3b-b10c-610b83df5f3f-workload-certs\") pod \"e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm\" (UID: \"8c34a7a9-02a7-4d3b-b10c-610b83df5f3f\") " pod="openshift-ingress/e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm" Jul 23 00:45:01.487554 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:01.487527 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c34a7a9-02a7-4d3b-b10c-610b83df5f3f-istiod-ca-cert\") pod \"e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm\" (UID: \"8c34a7a9-02a7-4d3b-b10c-610b83df5f3f\") " pod="openshift-ingress/e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm" Jul 23 00:45:01.489215 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:01.489192 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c34a7a9-02a7-4d3b-b10c-610b83df5f3f-istio-envoy\") pod \"e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm\" (UID: \"8c34a7a9-02a7-4d3b-b10c-610b83df5f3f\") " pod="openshift-ingress/e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm" Jul 23 00:45:01.489492 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:01.489471 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c34a7a9-02a7-4d3b-b10c-610b83df5f3f-istio-podinfo\") pod \"e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm\" (UID: \"8c34a7a9-02a7-4d3b-b10c-610b83df5f3f\") " pod="openshift-ingress/e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm" Jul 23 00:45:01.494803 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:01.494747 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c34a7a9-02a7-4d3b-b10c-610b83df5f3f-istio-token\") pod \"e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm\" (UID: \"8c34a7a9-02a7-4d3b-b10c-610b83df5f3f\") " pod="openshift-ingress/e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm" Jul 23 00:45:01.494994 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:01.494973 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkfmz\" (UniqueName: \"kubernetes.io/projected/8c34a7a9-02a7-4d3b-b10c-610b83df5f3f-kube-api-access-hkfmz\") pod \"e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm\" (UID: \"8c34a7a9-02a7-4d3b-b10c-610b83df5f3f\") " pod="openshift-ingress/e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm" Jul 23 00:45:01.565295 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:01.565254 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm" Jul 23 00:45:01.716517 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:01.716491 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm"] Jul 23 00:45:01.718747 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:45:01.718716 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c34a7a9_02a7_4d3b_b10c_610b83df5f3f.slice/crio-75a173f0788e9c59ae99e75f1a695e8a0f4ddaf59545407875f8c55700f5e2b2 WatchSource:0}: Error finding container 75a173f0788e9c59ae99e75f1a695e8a0f4ddaf59545407875f8c55700f5e2b2: Status 404 returned error can't find the container with id 75a173f0788e9c59ae99e75f1a695e8a0f4ddaf59545407875f8c55700f5e2b2 Jul 23 00:45:01.721044 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:01.721009 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:45:01.721128 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:01.721078 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:45:01.721128 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:01.721112 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:45:02.196248 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:02.196204 2562 generic.go:356] "Generic (PLEG): container finished" podID="60a410ac-a77e-47e3-8b3c-31bceacecc1d" containerID="c1b3fc1b77c853f5cce4786c9d4d41cf2ca46abdb4d0738b7d0adf96449ff02c" exitCode=0 Jul 23 00:45:02.196745 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:02.196285 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746125-s4rhn" event={"ID":"60a410ac-a77e-47e3-8b3c-31bceacecc1d","Type":"ContainerDied","Data":"c1b3fc1b77c853f5cce4786c9d4d41cf2ca46abdb4d0738b7d0adf96449ff02c"} Jul 23 00:45:02.197939 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:02.197913 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm" event={"ID":"8c34a7a9-02a7-4d3b-b10c-610b83df5f3f","Type":"ContainerStarted","Data":"22285bc278dc2490e914ce4add1f27b5b6b0bee78a175c692f13d2f1eda3ee03"} Jul 23 00:45:02.198058 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:02.197959 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm" event={"ID":"8c34a7a9-02a7-4d3b-b10c-610b83df5f3f","Type":"ContainerStarted","Data":"75a173f0788e9c59ae99e75f1a695e8a0f4ddaf59545407875f8c55700f5e2b2"} Jul 23 00:45:02.230553 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:02.230497 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm" podStartSLOduration=1.230478558 podStartE2EDuration="1.230478558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:45:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:45:02.229731072 +0000 UTC m=+2216.025237964" watchObservedRunningTime="2026-07-23 00:45:02.230478558 +0000 UTC m=+2216.025985418" Jul 23 00:45:02.565991 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:02.565950 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm" Jul 23 00:45:02.571274 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:02.571237 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm" Jul 23 00:45:03.203634 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:03.203594 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm" Jul 23 00:45:03.204739 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:03.204708 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm" Jul 23 00:45:03.354177 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:03.354147 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746125-s4rhn" Jul 23 00:45:04.210656 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:04.210627 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746125-s4rhn" Jul 23 00:45:04.211133 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:04.210665 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746125-s4rhn" event={"ID":"60a410ac-a77e-47e3-8b3c-31bceacecc1d","Type":"ContainerDied","Data":"f7dbfc578aa489d40d7a92cebc6e81c7f74af6e11b4bab1c47820e05e41ad458"} Jul 23 00:45:04.211133 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:04.210714 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7dbfc578aa489d40d7a92cebc6e81c7f74af6e11b4bab1c47820e05e41ad458" Jul 23 00:45:18.205089 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:18.204997 2562 patch_prober.go:28] interesting pod/e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Jul 23 00:45:18.205089 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:18.205069 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm" podUID="8c34a7a9-02a7-4d3b-b10c-610b83df5f3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Jul 23 00:45:18.293241 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:18.293207 2562 generic.go:356] "Generic (PLEG): container finished" podID="8c34a7a9-02a7-4d3b-b10c-610b83df5f3f" containerID="22285bc278dc2490e914ce4add1f27b5b6b0bee78a175c692f13d2f1eda3ee03" exitCode=0 Jul 23 00:45:18.293422 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:18.293288 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm" event={"ID":"8c34a7a9-02a7-4d3b-b10c-610b83df5f3f","Type":"ContainerDied","Data":"22285bc278dc2490e914ce4add1f27b5b6b0bee78a175c692f13d2f1eda3ee03"} Jul 23 00:45:18.293711 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:18.293697 2562 scope.go:117] "RemoveContainer" containerID="22285bc278dc2490e914ce4add1f27b5b6b0bee78a175c692f13d2f1eda3ee03" Jul 23 00:45:18.294729 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:18.294705 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:45:18.294959 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:18.294797 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:45:18.294959 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:18.294831 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:45:19.305598 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:19.305546 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm" event={"ID":"8c34a7a9-02a7-4d3b-b10c-610b83df5f3f","Type":"ContainerStarted","Data":"4f7579fda312339af84073867625ac01e5dfb3e5f09dabc0fd6e3d91c8b24abe"} Jul 23 00:45:19.565959 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:19.565867 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm" Jul 23 00:45:19.566118 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:19.566092 2562 patch_prober.go:28] interesting pod/e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Jul 23 00:45:19.566162 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:19.566142 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm" podUID="8c34a7a9-02a7-4d3b-b10c-610b83df5f3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Jul 23 00:45:20.566198 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:20.566158 2562 patch_prober.go:28] interesting pod/e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Jul 23 00:45:20.566616 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:20.566216 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm" podUID="8c34a7a9-02a7-4d3b-b10c-610b83df5f3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Jul 23 00:45:21.565882 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:21.565844 2562 patch_prober.go:28] interesting pod/e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Jul 23 00:45:21.566051 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:21.565905 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm" podUID="8c34a7a9-02a7-4d3b-b10c-610b83df5f3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Jul 23 00:45:22.566710 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:22.566668 2562 patch_prober.go:28] interesting pod/e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Jul 23 00:45:22.567173 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:22.566778 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm" podUID="8c34a7a9-02a7-4d3b-b10c-610b83df5f3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Jul 23 00:45:23.569939 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:23.569905 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm" Jul 23 00:45:23.570339 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:23.570176 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm" Jul 23 00:45:23.571060 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:23.571037 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm" Jul 23 00:45:37.346295 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:37.346210 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm"] Jul 23 00:45:37.346901 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:37.346443 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm" podUID="8c34a7a9-02a7-4d3b-b10c-610b83df5f3f" containerName="istio-proxy" containerID="cri-o://4f7579fda312339af84073867625ac01e5dfb3e5f09dabc0fd6e3d91c8b24abe" gracePeriod=30 Jul 23 00:45:38.573317 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:38.573273 2562 patch_prober.go:28] interesting pod/e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 23 00:45:38.573824 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:38.573357 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm" podUID="8c34a7a9-02a7-4d3b-b10c-610b83df5f3f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 23 00:45:42.600656 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:42.600624 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm" Jul 23 00:45:42.709689 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:42.709595 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c34a7a9-02a7-4d3b-b10c-610b83df5f3f-istio-token\") pod \"8c34a7a9-02a7-4d3b-b10c-610b83df5f3f\" (UID: \"8c34a7a9-02a7-4d3b-b10c-610b83df5f3f\") " Jul 23 00:45:42.709689 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:42.709665 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c34a7a9-02a7-4d3b-b10c-610b83df5f3f-istio-data\") pod \"8c34a7a9-02a7-4d3b-b10c-610b83df5f3f\" (UID: \"8c34a7a9-02a7-4d3b-b10c-610b83df5f3f\") " Jul 23 00:45:42.709955 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:42.709727 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hkfmz\" (UniqueName: \"kubernetes.io/projected/8c34a7a9-02a7-4d3b-b10c-610b83df5f3f-kube-api-access-hkfmz\") pod \"8c34a7a9-02a7-4d3b-b10c-610b83df5f3f\" (UID: \"8c34a7a9-02a7-4d3b-b10c-610b83df5f3f\") " Jul 23 00:45:42.709955 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:42.709784 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c34a7a9-02a7-4d3b-b10c-610b83df5f3f-workload-certs\") pod \"8c34a7a9-02a7-4d3b-b10c-610b83df5f3f\" (UID: \"8c34a7a9-02a7-4d3b-b10c-610b83df5f3f\") " Jul 23 00:45:42.709955 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:42.709813 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c34a7a9-02a7-4d3b-b10c-610b83df5f3f-istio-envoy\") pod \"8c34a7a9-02a7-4d3b-b10c-610b83df5f3f\" (UID: \"8c34a7a9-02a7-4d3b-b10c-610b83df5f3f\") " Jul 23 00:45:42.709955 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:42.709847 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c34a7a9-02a7-4d3b-b10c-610b83df5f3f-istio-podinfo\") pod \"8c34a7a9-02a7-4d3b-b10c-610b83df5f3f\" (UID: \"8c34a7a9-02a7-4d3b-b10c-610b83df5f3f\") " Jul 23 00:45:42.709955 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:42.709890 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c34a7a9-02a7-4d3b-b10c-610b83df5f3f-workload-socket\") pod \"8c34a7a9-02a7-4d3b-b10c-610b83df5f3f\" (UID: \"8c34a7a9-02a7-4d3b-b10c-610b83df5f3f\") " Jul 23 00:45:42.709955 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:42.709932 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c34a7a9-02a7-4d3b-b10c-610b83df5f3f-credential-socket\") pod \"8c34a7a9-02a7-4d3b-b10c-610b83df5f3f\" (UID: \"8c34a7a9-02a7-4d3b-b10c-610b83df5f3f\") " Jul 23 00:45:42.710255 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:42.709967 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c34a7a9-02a7-4d3b-b10c-610b83df5f3f-istiod-ca-cert\") pod \"8c34a7a9-02a7-4d3b-b10c-610b83df5f3f\" (UID: \"8c34a7a9-02a7-4d3b-b10c-610b83df5f3f\") " Jul 23 00:45:42.710255 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:42.710071 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c34a7a9-02a7-4d3b-b10c-610b83df5f3f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8c34a7a9-02a7-4d3b-b10c-610b83df5f3f" (UID: "8c34a7a9-02a7-4d3b-b10c-610b83df5f3f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:45:42.710360 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:42.710261 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c34a7a9-02a7-4d3b-b10c-610b83df5f3f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8c34a7a9-02a7-4d3b-b10c-610b83df5f3f" (UID: "8c34a7a9-02a7-4d3b-b10c-610b83df5f3f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:45:42.710418 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:42.710376 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c34a7a9-02a7-4d3b-b10c-610b83df5f3f-workload-certs\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:45:42.710418 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:42.710395 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c34a7a9-02a7-4d3b-b10c-610b83df5f3f-workload-socket\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:45:42.710539 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:42.710413 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c34a7a9-02a7-4d3b-b10c-610b83df5f3f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8c34a7a9-02a7-4d3b-b10c-610b83df5f3f" (UID: "8c34a7a9-02a7-4d3b-b10c-610b83df5f3f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:45:42.710615 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:42.710590 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c34a7a9-02a7-4d3b-b10c-610b83df5f3f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8c34a7a9-02a7-4d3b-b10c-610b83df5f3f" (UID: "8c34a7a9-02a7-4d3b-b10c-610b83df5f3f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:45:42.710615 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:42.710602 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c34a7a9-02a7-4d3b-b10c-610b83df5f3f-istio-data" (OuterVolumeSpecName: "istio-data") pod "8c34a7a9-02a7-4d3b-b10c-610b83df5f3f" (UID: "8c34a7a9-02a7-4d3b-b10c-610b83df5f3f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:45:42.712497 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:42.712465 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c34a7a9-02a7-4d3b-b10c-610b83df5f3f-istio-token" (OuterVolumeSpecName: "istio-token") pod "8c34a7a9-02a7-4d3b-b10c-610b83df5f3f" (UID: "8c34a7a9-02a7-4d3b-b10c-610b83df5f3f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:45:42.712619 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:42.712521 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c34a7a9-02a7-4d3b-b10c-610b83df5f3f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8c34a7a9-02a7-4d3b-b10c-610b83df5f3f" (UID: "8c34a7a9-02a7-4d3b-b10c-610b83df5f3f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 00:45:42.712619 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:42.712531 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c34a7a9-02a7-4d3b-b10c-610b83df5f3f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8c34a7a9-02a7-4d3b-b10c-610b83df5f3f" (UID: "8c34a7a9-02a7-4d3b-b10c-610b83df5f3f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:45:42.712619 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:42.712549 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c34a7a9-02a7-4d3b-b10c-610b83df5f3f-kube-api-access-hkfmz" (OuterVolumeSpecName: "kube-api-access-hkfmz") pod "8c34a7a9-02a7-4d3b-b10c-610b83df5f3f" (UID: "8c34a7a9-02a7-4d3b-b10c-610b83df5f3f"). InnerVolumeSpecName "kube-api-access-hkfmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:45:42.811746 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:42.811709 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c34a7a9-02a7-4d3b-b10c-610b83df5f3f-istio-envoy\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:45:42.811746 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:42.811743 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c34a7a9-02a7-4d3b-b10c-610b83df5f3f-istio-podinfo\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:45:42.811746 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:42.811752 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c34a7a9-02a7-4d3b-b10c-610b83df5f3f-credential-socket\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:45:42.812036 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:42.811793 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c34a7a9-02a7-4d3b-b10c-610b83df5f3f-istiod-ca-cert\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:45:42.812036 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:42.811802 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c34a7a9-02a7-4d3b-b10c-610b83df5f3f-istio-token\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:45:42.812036 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:42.811811 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c34a7a9-02a7-4d3b-b10c-610b83df5f3f-istio-data\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:45:42.812036 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:42.811819 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkfmz\" (UniqueName: \"kubernetes.io/projected/8c34a7a9-02a7-4d3b-b10c-610b83df5f3f-kube-api-access-hkfmz\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:45:43.453334 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:43.453301 2562 generic.go:356] "Generic (PLEG): container finished" podID="8c34a7a9-02a7-4d3b-b10c-610b83df5f3f" containerID="4f7579fda312339af84073867625ac01e5dfb3e5f09dabc0fd6e3d91c8b24abe" exitCode=0 Jul 23 00:45:43.453519 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:43.453346 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm" Jul 23 00:45:43.453519 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:43.453350 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm" event={"ID":"8c34a7a9-02a7-4d3b-b10c-610b83df5f3f","Type":"ContainerDied","Data":"4f7579fda312339af84073867625ac01e5dfb3e5f09dabc0fd6e3d91c8b24abe"} Jul 23 00:45:43.453519 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:43.453496 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm" event={"ID":"8c34a7a9-02a7-4d3b-b10c-610b83df5f3f","Type":"ContainerDied","Data":"75a173f0788e9c59ae99e75f1a695e8a0f4ddaf59545407875f8c55700f5e2b2"} Jul 23 00:45:43.453645 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:43.453530 2562 scope.go:117] "RemoveContainer" containerID="4f7579fda312339af84073867625ac01e5dfb3e5f09dabc0fd6e3d91c8b24abe" Jul 23 00:45:43.463192 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:43.463172 2562 scope.go:117] "RemoveContainer" containerID="22285bc278dc2490e914ce4add1f27b5b6b0bee78a175c692f13d2f1eda3ee03" Jul 23 00:45:43.472207 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:43.472082 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm"] Jul 23 00:45:43.472294 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:43.472272 2562 scope.go:117] "RemoveContainer" containerID="4f7579fda312339af84073867625ac01e5dfb3e5f09dabc0fd6e3d91c8b24abe" Jul 23 00:45:43.472599 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:45:43.472580 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f7579fda312339af84073867625ac01e5dfb3e5f09dabc0fd6e3d91c8b24abe\": container with ID starting with 4f7579fda312339af84073867625ac01e5dfb3e5f09dabc0fd6e3d91c8b24abe not found: ID does not exist" containerID="4f7579fda312339af84073867625ac01e5dfb3e5f09dabc0fd6e3d91c8b24abe" Jul 23 00:45:43.472671 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:43.472611 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f7579fda312339af84073867625ac01e5dfb3e5f09dabc0fd6e3d91c8b24abe"} err="failed to get container status \"4f7579fda312339af84073867625ac01e5dfb3e5f09dabc0fd6e3d91c8b24abe\": rpc error: code = NotFound desc = could not find container \"4f7579fda312339af84073867625ac01e5dfb3e5f09dabc0fd6e3d91c8b24abe\": container with ID starting with 4f7579fda312339af84073867625ac01e5dfb3e5f09dabc0fd6e3d91c8b24abe not found: ID does not exist" Jul 23 00:45:43.472671 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:43.472638 2562 scope.go:117] "RemoveContainer" containerID="22285bc278dc2490e914ce4add1f27b5b6b0bee78a175c692f13d2f1eda3ee03" Jul 23 00:45:43.472948 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:45:43.472930 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22285bc278dc2490e914ce4add1f27b5b6b0bee78a175c692f13d2f1eda3ee03\": container with ID starting with 22285bc278dc2490e914ce4add1f27b5b6b0bee78a175c692f13d2f1eda3ee03 not found: ID does not exist" containerID="22285bc278dc2490e914ce4add1f27b5b6b0bee78a175c692f13d2f1eda3ee03" Jul 23 00:45:43.472993 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:43.472955 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22285bc278dc2490e914ce4add1f27b5b6b0bee78a175c692f13d2f1eda3ee03"} err="failed to get container status \"22285bc278dc2490e914ce4add1f27b5b6b0bee78a175c692f13d2f1eda3ee03\": rpc error: code = NotFound desc = could not find container \"22285bc278dc2490e914ce4add1f27b5b6b0bee78a175c692f13d2f1eda3ee03\": container with ID starting with 22285bc278dc2490e914ce4add1f27b5b6b0bee78a175c692f13d2f1eda3ee03 not found: ID does not exist" Jul 23 00:45:43.477425 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:43.477401 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c65690b4-openshift-default-6675975bdc-xprvm"] Jul 23 00:45:45.006829 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:45.006798 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c34a7a9-02a7-4d3b-b10c-610b83df5f3f" path="/var/lib/kubelet/pods/8c34a7a9-02a7-4d3b-b10c-610b83df5f3f/volumes" Jul 23 00:45:56.940122 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:56.940088 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts"] Jul 23 00:45:56.940525 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:56.940508 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c34a7a9-02a7-4d3b-b10c-610b83df5f3f" containerName="istio-proxy" Jul 23 00:45:56.940583 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:56.940529 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c34a7a9-02a7-4d3b-b10c-610b83df5f3f" containerName="istio-proxy" Jul 23 00:45:56.940583 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:56.940546 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60a410ac-a77e-47e3-8b3c-31bceacecc1d" containerName="cleanup" Jul 23 00:45:56.940583 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:56.940552 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="60a410ac-a77e-47e3-8b3c-31bceacecc1d" containerName="cleanup" Jul 23 00:45:56.940716 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:56.940625 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c34a7a9-02a7-4d3b-b10c-610b83df5f3f" containerName="istio-proxy" Jul 23 00:45:56.940716 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:56.940636 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c34a7a9-02a7-4d3b-b10c-610b83df5f3f" containerName="istio-proxy" Jul 23 00:45:56.940716 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:56.940648 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="60a410ac-a77e-47e3-8b3c-31bceacecc1d" containerName="cleanup" Jul 23 00:45:56.945247 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:56.945222 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts" Jul 23 00:45:56.948245 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:56.948222 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-76cb26bd-openshift-default-dockercfg-6bwrt\"" Jul 23 00:45:56.955192 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:56.955164 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts"] Jul 23 00:45:57.056913 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:57.056869 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/befef356-b344-4dc8-b284-0ed1af4fb6f6-istio-envoy\") pod \"e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts\" (UID: \"befef356-b344-4dc8-b284-0ed1af4fb6f6\") " pod="openshift-ingress/e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts" Jul 23 00:45:57.057120 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:57.056993 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/befef356-b344-4dc8-b284-0ed1af4fb6f6-istiod-ca-cert\") pod \"e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts\" (UID: \"befef356-b344-4dc8-b284-0ed1af4fb6f6\") " pod="openshift-ingress/e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts" Jul 23 00:45:57.057120 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:57.057037 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/befef356-b344-4dc8-b284-0ed1af4fb6f6-credential-socket\") pod \"e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts\" (UID: \"befef356-b344-4dc8-b284-0ed1af4fb6f6\") " pod="openshift-ingress/e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts" Jul 23 00:45:57.057120 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:57.057076 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/befef356-b344-4dc8-b284-0ed1af4fb6f6-istio-podinfo\") pod \"e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts\" (UID: \"befef356-b344-4dc8-b284-0ed1af4fb6f6\") " pod="openshift-ingress/e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts" Jul 23 00:45:57.057120 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:57.057099 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/befef356-b344-4dc8-b284-0ed1af4fb6f6-istio-token\") pod \"e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts\" (UID: \"befef356-b344-4dc8-b284-0ed1af4fb6f6\") " pod="openshift-ingress/e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts" Jul 23 00:45:57.057329 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:57.057163 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/befef356-b344-4dc8-b284-0ed1af4fb6f6-workload-socket\") pod \"e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts\" (UID: \"befef356-b344-4dc8-b284-0ed1af4fb6f6\") " pod="openshift-ingress/e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts" Jul 23 00:45:57.057329 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:57.057194 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/befef356-b344-4dc8-b284-0ed1af4fb6f6-istio-data\") pod \"e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts\" (UID: \"befef356-b344-4dc8-b284-0ed1af4fb6f6\") " pod="openshift-ingress/e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts" Jul 23 00:45:57.057329 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:57.057226 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/befef356-b344-4dc8-b284-0ed1af4fb6f6-workload-certs\") pod \"e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts\" (UID: \"befef356-b344-4dc8-b284-0ed1af4fb6f6\") " pod="openshift-ingress/e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts" Jul 23 00:45:57.057329 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:57.057274 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w848x\" (UniqueName: \"kubernetes.io/projected/befef356-b344-4dc8-b284-0ed1af4fb6f6-kube-api-access-w848x\") pod \"e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts\" (UID: \"befef356-b344-4dc8-b284-0ed1af4fb6f6\") " pod="openshift-ingress/e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts" Jul 23 00:45:57.157820 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:57.157788 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/befef356-b344-4dc8-b284-0ed1af4fb6f6-istio-podinfo\") pod \"e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts\" (UID: \"befef356-b344-4dc8-b284-0ed1af4fb6f6\") " pod="openshift-ingress/e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts" Jul 23 00:45:57.158028 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:57.157828 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/befef356-b344-4dc8-b284-0ed1af4fb6f6-istio-token\") pod \"e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts\" (UID: \"befef356-b344-4dc8-b284-0ed1af4fb6f6\") " pod="openshift-ingress/e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts" Jul 23 00:45:57.158028 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:57.157891 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/befef356-b344-4dc8-b284-0ed1af4fb6f6-workload-socket\") pod \"e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts\" (UID: \"befef356-b344-4dc8-b284-0ed1af4fb6f6\") " pod="openshift-ingress/e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts" Jul 23 00:45:57.158028 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:57.157918 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/befef356-b344-4dc8-b284-0ed1af4fb6f6-istio-data\") pod \"e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts\" (UID: \"befef356-b344-4dc8-b284-0ed1af4fb6f6\") " pod="openshift-ingress/e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts" Jul 23 00:45:57.158196 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:57.158114 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/befef356-b344-4dc8-b284-0ed1af4fb6f6-workload-certs\") pod \"e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts\" (UID: \"befef356-b344-4dc8-b284-0ed1af4fb6f6\") " pod="openshift-ingress/e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts" Jul 23 00:45:57.158196 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:57.158153 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w848x\" (UniqueName: \"kubernetes.io/projected/befef356-b344-4dc8-b284-0ed1af4fb6f6-kube-api-access-w848x\") pod \"e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts\" (UID: \"befef356-b344-4dc8-b284-0ed1af4fb6f6\") " pod="openshift-ingress/e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts" Jul 23 00:45:57.158309 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:57.158205 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/befef356-b344-4dc8-b284-0ed1af4fb6f6-istio-envoy\") pod \"e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts\" (UID: \"befef356-b344-4dc8-b284-0ed1af4fb6f6\") " pod="openshift-ingress/e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts" Jul 23 00:45:57.158309 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:57.158297 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/befef356-b344-4dc8-b284-0ed1af4fb6f6-istiod-ca-cert\") pod \"e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts\" (UID: \"befef356-b344-4dc8-b284-0ed1af4fb6f6\") " pod="openshift-ingress/e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts" Jul 23 00:45:57.158423 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:57.158338 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/befef356-b344-4dc8-b284-0ed1af4fb6f6-credential-socket\") pod \"e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts\" (UID: \"befef356-b344-4dc8-b284-0ed1af4fb6f6\") " pod="openshift-ingress/e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts" Jul 23 00:45:57.158423 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:57.158356 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/befef356-b344-4dc8-b284-0ed1af4fb6f6-istio-data\") pod \"e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts\" (UID: \"befef356-b344-4dc8-b284-0ed1af4fb6f6\") " pod="openshift-ingress/e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts" Jul 23 00:45:57.158532 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:57.158430 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/befef356-b344-4dc8-b284-0ed1af4fb6f6-workload-socket\") pod \"e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts\" (UID: \"befef356-b344-4dc8-b284-0ed1af4fb6f6\") " pod="openshift-ingress/e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts" Jul 23 00:45:57.158532 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:57.158477 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/befef356-b344-4dc8-b284-0ed1af4fb6f6-workload-certs\") pod \"e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts\" (UID: \"befef356-b344-4dc8-b284-0ed1af4fb6f6\") " pod="openshift-ingress/e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts" Jul 23 00:45:57.158718 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:57.158700 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/befef356-b344-4dc8-b284-0ed1af4fb6f6-credential-socket\") pod \"e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts\" (UID: \"befef356-b344-4dc8-b284-0ed1af4fb6f6\") " pod="openshift-ingress/e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts" Jul 23 00:45:57.158963 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:57.158946 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/befef356-b344-4dc8-b284-0ed1af4fb6f6-istiod-ca-cert\") pod \"e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts\" (UID: \"befef356-b344-4dc8-b284-0ed1af4fb6f6\") " pod="openshift-ingress/e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts" Jul 23 00:45:57.160295 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:57.160275 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/befef356-b344-4dc8-b284-0ed1af4fb6f6-istio-envoy\") pod \"e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts\" (UID: \"befef356-b344-4dc8-b284-0ed1af4fb6f6\") " pod="openshift-ingress/e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts" Jul 23 00:45:57.160402 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:57.160378 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/befef356-b344-4dc8-b284-0ed1af4fb6f6-istio-podinfo\") pod \"e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts\" (UID: \"befef356-b344-4dc8-b284-0ed1af4fb6f6\") " pod="openshift-ingress/e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts" Jul 23 00:45:57.166035 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:57.166010 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/befef356-b344-4dc8-b284-0ed1af4fb6f6-istio-token\") pod \"e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts\" (UID: \"befef356-b344-4dc8-b284-0ed1af4fb6f6\") " pod="openshift-ingress/e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts" Jul 23 00:45:57.166134 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:57.166065 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w848x\" (UniqueName: \"kubernetes.io/projected/befef356-b344-4dc8-b284-0ed1af4fb6f6-kube-api-access-w848x\") pod \"e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts\" (UID: \"befef356-b344-4dc8-b284-0ed1af4fb6f6\") " pod="openshift-ingress/e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts" Jul 23 00:45:57.258989 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:57.258898 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts" Jul 23 00:45:57.402864 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:57.402822 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts"] Jul 23 00:45:57.406172 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:45:57.406135 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbefef356_b344_4dc8_b284_0ed1af4fb6f6.slice/crio-3bffdafdc318d531c762830c5ecf5384eb784df1c9894fdbaab4ac69dae4aa5e WatchSource:0}: Error finding container 3bffdafdc318d531c762830c5ecf5384eb784df1c9894fdbaab4ac69dae4aa5e: Status 404 returned error can't find the container with id 3bffdafdc318d531c762830c5ecf5384eb784df1c9894fdbaab4ac69dae4aa5e Jul 23 00:45:57.408354 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:57.408322 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:45:57.408448 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:57.408391 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:45:57.408448 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:57.408421 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:45:57.543005 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:57.542966 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts" event={"ID":"befef356-b344-4dc8-b284-0ed1af4fb6f6","Type":"ContainerStarted","Data":"998d0413eabf2223321bf38ff3e580396d6fc990197ecf814209af1aada4bb8d"} Jul 23 00:45:57.543206 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:57.543034 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts" event={"ID":"befef356-b344-4dc8-b284-0ed1af4fb6f6","Type":"ContainerStarted","Data":"3bffdafdc318d531c762830c5ecf5384eb784df1c9894fdbaab4ac69dae4aa5e"} Jul 23 00:45:57.562690 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:57.562628 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts" podStartSLOduration=1.562606859 podStartE2EDuration="1.562606859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:45:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:45:57.560940517 +0000 UTC m=+2271.356447376" watchObservedRunningTime="2026-07-23 00:45:57.562606859 +0000 UTC m=+2271.358113721" Jul 23 00:45:58.259264 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:58.259225 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts" Jul 23 00:45:58.264341 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:58.264308 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts" Jul 23 00:45:58.548829 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:58.548794 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts" Jul 23 00:45:58.549880 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:45:58.549861 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts" Jul 23 00:46:14.590028 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:46:14.589977 2562 patch_prober.go:28] interesting pod/e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 23 00:46:14.590519 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:46:14.590045 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts" podUID="befef356-b344-4dc8-b284-0ed1af4fb6f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 23 00:46:15.657638 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:46:15.657599 2562 generic.go:356] "Generic (PLEG): container finished" podID="befef356-b344-4dc8-b284-0ed1af4fb6f6" containerID="998d0413eabf2223321bf38ff3e580396d6fc990197ecf814209af1aada4bb8d" exitCode=0 Jul 23 00:46:15.658131 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:46:15.657658 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts" event={"ID":"befef356-b344-4dc8-b284-0ed1af4fb6f6","Type":"ContainerDied","Data":"998d0413eabf2223321bf38ff3e580396d6fc990197ecf814209af1aada4bb8d"} Jul 23 00:46:15.658131 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:46:15.658073 2562 scope.go:117] "RemoveContainer" containerID="998d0413eabf2223321bf38ff3e580396d6fc990197ecf814209af1aada4bb8d" Jul 23 00:46:15.659043 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:46:15.659013 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:46:15.659151 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:46:15.659112 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:46:15.659221 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:46:15.659163 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:46:16.665659 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:46:16.665620 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts" event={"ID":"befef356-b344-4dc8-b284-0ed1af4fb6f6","Type":"ContainerStarted","Data":"68d1a6d19a2cdb2d5b8fe5e7c48ed3fe6363a2f469fabc4675aeff8eec84d32e"} Jul 23 00:46:17.259410 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:46:17.259363 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts" Jul 23 00:46:17.259688 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:46:17.259662 2562 patch_prober.go:28] interesting pod/e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 23 00:46:17.259812 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:46:17.259737 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts" podUID="befef356-b344-4dc8-b284-0ed1af4fb6f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 23 00:46:18.259894 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:46:18.259851 2562 patch_prober.go:28] interesting pod/e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 23 00:46:18.260300 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:46:18.259909 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts" podUID="befef356-b344-4dc8-b284-0ed1af4fb6f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 23 00:46:19.259520 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:46:19.259483 2562 patch_prober.go:28] interesting pod/e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 23 00:46:19.259707 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:46:19.259539 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts" podUID="befef356-b344-4dc8-b284-0ed1af4fb6f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 23 00:46:20.259361 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:46:20.259319 2562 patch_prober.go:28] interesting pod/e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 23 00:46:20.259891 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:46:20.259382 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts" podUID="befef356-b344-4dc8-b284-0ed1af4fb6f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 23 00:46:21.263052 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:46:21.263015 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts" Jul 23 00:46:21.263450 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:46:21.263410 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts" Jul 23 00:46:21.264051 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:46:21.264030 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts" Jul 23 00:48:03.726422 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:03.726384 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts"] Jul 23 00:48:03.727023 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:03.726639 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts" podUID="befef356-b344-4dc8-b284-0ed1af4fb6f6" containerName="istio-proxy" containerID="cri-o://68d1a6d19a2cdb2d5b8fe5e7c48ed3fe6363a2f469fabc4675aeff8eec84d32e" gracePeriod=30 Jul 23 00:48:06.265897 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:06.265860 2562 patch_prober.go:28] interesting pod/e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 23 00:48:06.266298 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:06.265919 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts" podUID="befef356-b344-4dc8-b284-0ed1af4fb6f6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 23 00:48:07.077622 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:07.072620 2562 scope.go:117] "RemoveContainer" containerID="6fdb47198cc8e12afdd5d9ed860e793b65886eb07f580a86fd0f9ac30c9c4ba6" Jul 23 00:48:08.987023 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:08.986952 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts" Jul 23 00:48:09.028175 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:09.028142 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/befef356-b344-4dc8-b284-0ed1af4fb6f6-istio-podinfo\") pod \"befef356-b344-4dc8-b284-0ed1af4fb6f6\" (UID: \"befef356-b344-4dc8-b284-0ed1af4fb6f6\") " Jul 23 00:48:09.028364 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:09.028190 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/befef356-b344-4dc8-b284-0ed1af4fb6f6-istio-token\") pod \"befef356-b344-4dc8-b284-0ed1af4fb6f6\" (UID: \"befef356-b344-4dc8-b284-0ed1af4fb6f6\") " Jul 23 00:48:09.028364 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:09.028214 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/befef356-b344-4dc8-b284-0ed1af4fb6f6-credential-socket\") pod \"befef356-b344-4dc8-b284-0ed1af4fb6f6\" (UID: \"befef356-b344-4dc8-b284-0ed1af4fb6f6\") " Jul 23 00:48:09.028364 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:09.028245 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w848x\" (UniqueName: \"kubernetes.io/projected/befef356-b344-4dc8-b284-0ed1af4fb6f6-kube-api-access-w848x\") pod \"befef356-b344-4dc8-b284-0ed1af4fb6f6\" (UID: \"befef356-b344-4dc8-b284-0ed1af4fb6f6\") " Jul 23 00:48:09.028364 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:09.028290 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/befef356-b344-4dc8-b284-0ed1af4fb6f6-workload-certs\") pod \"befef356-b344-4dc8-b284-0ed1af4fb6f6\" (UID: \"befef356-b344-4dc8-b284-0ed1af4fb6f6\") " Jul 23 00:48:09.028364 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:09.028353 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/befef356-b344-4dc8-b284-0ed1af4fb6f6-istio-data\") pod \"befef356-b344-4dc8-b284-0ed1af4fb6f6\" (UID: \"befef356-b344-4dc8-b284-0ed1af4fb6f6\") " Jul 23 00:48:09.028637 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:09.028376 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/befef356-b344-4dc8-b284-0ed1af4fb6f6-istio-envoy\") pod \"befef356-b344-4dc8-b284-0ed1af4fb6f6\" (UID: \"befef356-b344-4dc8-b284-0ed1af4fb6f6\") " Jul 23 00:48:09.028637 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:09.028440 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/befef356-b344-4dc8-b284-0ed1af4fb6f6-istiod-ca-cert\") pod \"befef356-b344-4dc8-b284-0ed1af4fb6f6\" (UID: \"befef356-b344-4dc8-b284-0ed1af4fb6f6\") " Jul 23 00:48:09.028637 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:09.028472 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/befef356-b344-4dc8-b284-0ed1af4fb6f6-workload-socket\") pod \"befef356-b344-4dc8-b284-0ed1af4fb6f6\" (UID: \"befef356-b344-4dc8-b284-0ed1af4fb6f6\") " Jul 23 00:48:09.028637 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:09.028566 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/befef356-b344-4dc8-b284-0ed1af4fb6f6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "befef356-b344-4dc8-b284-0ed1af4fb6f6" (UID: "befef356-b344-4dc8-b284-0ed1af4fb6f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:48:09.028637 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:09.028581 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/befef356-b344-4dc8-b284-0ed1af4fb6f6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "befef356-b344-4dc8-b284-0ed1af4fb6f6" (UID: "befef356-b344-4dc8-b284-0ed1af4fb6f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:48:09.028982 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:09.028797 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/befef356-b344-4dc8-b284-0ed1af4fb6f6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "befef356-b344-4dc8-b284-0ed1af4fb6f6" (UID: "befef356-b344-4dc8-b284-0ed1af4fb6f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:48:09.028982 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:09.028901 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/befef356-b344-4dc8-b284-0ed1af4fb6f6-workload-socket\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:48:09.028982 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:09.028919 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/befef356-b344-4dc8-b284-0ed1af4fb6f6-credential-socket\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:48:09.028982 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:09.028932 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/befef356-b344-4dc8-b284-0ed1af4fb6f6-workload-certs\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:48:09.029188 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:09.029101 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/befef356-b344-4dc8-b284-0ed1af4fb6f6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "befef356-b344-4dc8-b284-0ed1af4fb6f6" (UID: "befef356-b344-4dc8-b284-0ed1af4fb6f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:48:09.029373 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:09.029351 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/befef356-b344-4dc8-b284-0ed1af4fb6f6-istio-data" (OuterVolumeSpecName: "istio-data") pod "befef356-b344-4dc8-b284-0ed1af4fb6f6" (UID: "befef356-b344-4dc8-b284-0ed1af4fb6f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:48:09.030664 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:09.030631 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/befef356-b344-4dc8-b284-0ed1af4fb6f6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "befef356-b344-4dc8-b284-0ed1af4fb6f6" (UID: "befef356-b344-4dc8-b284-0ed1af4fb6f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:48:09.030995 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:09.030972 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/befef356-b344-4dc8-b284-0ed1af4fb6f6-kube-api-access-w848x" (OuterVolumeSpecName: "kube-api-access-w848x") pod "befef356-b344-4dc8-b284-0ed1af4fb6f6" (UID: "befef356-b344-4dc8-b284-0ed1af4fb6f6"). InnerVolumeSpecName "kube-api-access-w848x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:48:09.031079 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:09.030978 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/befef356-b344-4dc8-b284-0ed1af4fb6f6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "befef356-b344-4dc8-b284-0ed1af4fb6f6" (UID: "befef356-b344-4dc8-b284-0ed1af4fb6f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 00:48:09.031079 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:09.030990 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/befef356-b344-4dc8-b284-0ed1af4fb6f6-istio-token" (OuterVolumeSpecName: "istio-token") pod "befef356-b344-4dc8-b284-0ed1af4fb6f6" (UID: "befef356-b344-4dc8-b284-0ed1af4fb6f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:48:09.129438 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:09.129399 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/befef356-b344-4dc8-b284-0ed1af4fb6f6-istio-data\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:48:09.129438 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:09.129434 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/befef356-b344-4dc8-b284-0ed1af4fb6f6-istio-envoy\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:48:09.129438 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:09.129443 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/befef356-b344-4dc8-b284-0ed1af4fb6f6-istiod-ca-cert\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:48:09.129438 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:09.129452 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/befef356-b344-4dc8-b284-0ed1af4fb6f6-istio-podinfo\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:48:09.129728 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:09.129459 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/befef356-b344-4dc8-b284-0ed1af4fb6f6-istio-token\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:48:09.129728 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:09.129469 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w848x\" (UniqueName: \"kubernetes.io/projected/befef356-b344-4dc8-b284-0ed1af4fb6f6-kube-api-access-w848x\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:48:09.312027 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:09.311992 2562 generic.go:356] "Generic (PLEG): container finished" podID="befef356-b344-4dc8-b284-0ed1af4fb6f6" containerID="68d1a6d19a2cdb2d5b8fe5e7c48ed3fe6363a2f469fabc4675aeff8eec84d32e" exitCode=0 Jul 23 00:48:09.312201 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:09.312041 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts" Jul 23 00:48:09.312201 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:09.312070 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts" event={"ID":"befef356-b344-4dc8-b284-0ed1af4fb6f6","Type":"ContainerDied","Data":"68d1a6d19a2cdb2d5b8fe5e7c48ed3fe6363a2f469fabc4675aeff8eec84d32e"} Jul 23 00:48:09.312201 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:09.312125 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts" event={"ID":"befef356-b344-4dc8-b284-0ed1af4fb6f6","Type":"ContainerDied","Data":"3bffdafdc318d531c762830c5ecf5384eb784df1c9894fdbaab4ac69dae4aa5e"} Jul 23 00:48:09.312201 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:09.312147 2562 scope.go:117] "RemoveContainer" containerID="68d1a6d19a2cdb2d5b8fe5e7c48ed3fe6363a2f469fabc4675aeff8eec84d32e" Jul 23 00:48:09.322460 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:09.322438 2562 scope.go:117] "RemoveContainer" containerID="998d0413eabf2223321bf38ff3e580396d6fc990197ecf814209af1aada4bb8d" Jul 23 00:48:09.331649 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:09.331620 2562 scope.go:117] "RemoveContainer" containerID="68d1a6d19a2cdb2d5b8fe5e7c48ed3fe6363a2f469fabc4675aeff8eec84d32e" Jul 23 00:48:09.332013 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:48:09.331989 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68d1a6d19a2cdb2d5b8fe5e7c48ed3fe6363a2f469fabc4675aeff8eec84d32e\": container with ID starting with 68d1a6d19a2cdb2d5b8fe5e7c48ed3fe6363a2f469fabc4675aeff8eec84d32e not found: ID does not exist" containerID="68d1a6d19a2cdb2d5b8fe5e7c48ed3fe6363a2f469fabc4675aeff8eec84d32e" Jul 23 00:48:09.332107 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:09.332021 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68d1a6d19a2cdb2d5b8fe5e7c48ed3fe6363a2f469fabc4675aeff8eec84d32e"} err="failed to get container status \"68d1a6d19a2cdb2d5b8fe5e7c48ed3fe6363a2f469fabc4675aeff8eec84d32e\": rpc error: code = NotFound desc = could not find container \"68d1a6d19a2cdb2d5b8fe5e7c48ed3fe6363a2f469fabc4675aeff8eec84d32e\": container with ID starting with 68d1a6d19a2cdb2d5b8fe5e7c48ed3fe6363a2f469fabc4675aeff8eec84d32e not found: ID does not exist" Jul 23 00:48:09.332107 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:09.332041 2562 scope.go:117] "RemoveContainer" containerID="998d0413eabf2223321bf38ff3e580396d6fc990197ecf814209af1aada4bb8d" Jul 23 00:48:09.332304 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:48:09.332280 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"998d0413eabf2223321bf38ff3e580396d6fc990197ecf814209af1aada4bb8d\": container with ID starting with 998d0413eabf2223321bf38ff3e580396d6fc990197ecf814209af1aada4bb8d not found: ID does not exist" containerID="998d0413eabf2223321bf38ff3e580396d6fc990197ecf814209af1aada4bb8d" Jul 23 00:48:09.332349 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:09.332315 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"998d0413eabf2223321bf38ff3e580396d6fc990197ecf814209af1aada4bb8d"} err="failed to get container status \"998d0413eabf2223321bf38ff3e580396d6fc990197ecf814209af1aada4bb8d\": rpc error: code = NotFound desc = could not find container \"998d0413eabf2223321bf38ff3e580396d6fc990197ecf814209af1aada4bb8d\": container with ID starting with 998d0413eabf2223321bf38ff3e580396d6fc990197ecf814209af1aada4bb8d not found: ID does not exist" Jul 23 00:48:09.338030 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:09.337990 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts"] Jul 23 00:48:09.340345 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:09.340319 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-76cb26bd-openshift-default-79b88db959-9s8ts"] Jul 23 00:48:11.007667 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:11.007624 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="befef356-b344-4dc8-b284-0ed1af4fb6f6" path="/var/lib/kubelet/pods/befef356-b344-4dc8-b284-0ed1af4fb6f6/volumes" Jul 23 00:48:59.168493 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:59.168404 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx"] Jul 23 00:48:59.169015 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:59.168847 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="befef356-b344-4dc8-b284-0ed1af4fb6f6" containerName="istio-proxy" Jul 23 00:48:59.169015 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:59.168861 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="befef356-b344-4dc8-b284-0ed1af4fb6f6" containerName="istio-proxy" Jul 23 00:48:59.169015 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:59.168872 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c34a7a9-02a7-4d3b-b10c-610b83df5f3f" containerName="istio-proxy" Jul 23 00:48:59.169015 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:59.168878 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c34a7a9-02a7-4d3b-b10c-610b83df5f3f" containerName="istio-proxy" Jul 23 00:48:59.169015 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:59.168886 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="befef356-b344-4dc8-b284-0ed1af4fb6f6" containerName="istio-proxy" Jul 23 00:48:59.169015 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:59.168891 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="befef356-b344-4dc8-b284-0ed1af4fb6f6" containerName="istio-proxy" Jul 23 00:48:59.169015 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:59.168959 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="befef356-b344-4dc8-b284-0ed1af4fb6f6" containerName="istio-proxy" Jul 23 00:48:59.169015 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:59.168968 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="befef356-b344-4dc8-b284-0ed1af4fb6f6" containerName="istio-proxy" Jul 23 00:48:59.171049 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:59.171032 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx" Jul 23 00:48:59.173991 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:59.173970 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-e5bc11-openshift-default-dockercfg-57sdr\"" Jul 23 00:48:59.183799 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:59.183751 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx"] Jul 23 00:48:59.210216 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:59.210164 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d216ac9-957c-46fa-9fa8-b612bda28053-istio-data\") pod \"e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx\" (UID: \"2d216ac9-957c-46fa-9fa8-b612bda28053\") " pod="openshift-ingress/e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx" Jul 23 00:48:59.210216 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:59.210214 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d216ac9-957c-46fa-9fa8-b612bda28053-istio-envoy\") pod \"e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx\" (UID: \"2d216ac9-957c-46fa-9fa8-b612bda28053\") " pod="openshift-ingress/e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx" Jul 23 00:48:59.210457 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:59.210286 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tkg6\" (UniqueName: \"kubernetes.io/projected/2d216ac9-957c-46fa-9fa8-b612bda28053-kube-api-access-5tkg6\") pod \"e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx\" (UID: \"2d216ac9-957c-46fa-9fa8-b612bda28053\") " pod="openshift-ingress/e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx" Jul 23 00:48:59.210457 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:59.210358 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d216ac9-957c-46fa-9fa8-b612bda28053-istio-podinfo\") pod \"e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx\" (UID: \"2d216ac9-957c-46fa-9fa8-b612bda28053\") " pod="openshift-ingress/e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx" Jul 23 00:48:59.210457 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:59.210401 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d216ac9-957c-46fa-9fa8-b612bda28053-workload-socket\") pod \"e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx\" (UID: \"2d216ac9-957c-46fa-9fa8-b612bda28053\") " pod="openshift-ingress/e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx" Jul 23 00:48:59.210457 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:59.210423 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d216ac9-957c-46fa-9fa8-b612bda28053-workload-certs\") pod \"e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx\" (UID: \"2d216ac9-957c-46fa-9fa8-b612bda28053\") " pod="openshift-ingress/e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx" Jul 23 00:48:59.210605 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:59.210495 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d216ac9-957c-46fa-9fa8-b612bda28053-istiod-ca-cert\") pod \"e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx\" (UID: \"2d216ac9-957c-46fa-9fa8-b612bda28053\") " pod="openshift-ingress/e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx" Jul 23 00:48:59.210605 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:59.210532 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d216ac9-957c-46fa-9fa8-b612bda28053-istio-token\") pod \"e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx\" (UID: \"2d216ac9-957c-46fa-9fa8-b612bda28053\") " pod="openshift-ingress/e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx" Jul 23 00:48:59.210605 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:59.210558 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d216ac9-957c-46fa-9fa8-b612bda28053-credential-socket\") pod \"e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx\" (UID: \"2d216ac9-957c-46fa-9fa8-b612bda28053\") " pod="openshift-ingress/e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx" Jul 23 00:48:59.311399 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:59.311355 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d216ac9-957c-46fa-9fa8-b612bda28053-istiod-ca-cert\") pod \"e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx\" (UID: \"2d216ac9-957c-46fa-9fa8-b612bda28053\") " pod="openshift-ingress/e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx" Jul 23 00:48:59.311399 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:59.311414 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d216ac9-957c-46fa-9fa8-b612bda28053-istio-token\") pod \"e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx\" (UID: \"2d216ac9-957c-46fa-9fa8-b612bda28053\") " pod="openshift-ingress/e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx" Jul 23 00:48:59.311669 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:59.311439 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d216ac9-957c-46fa-9fa8-b612bda28053-credential-socket\") pod \"e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx\" (UID: \"2d216ac9-957c-46fa-9fa8-b612bda28053\") " pod="openshift-ingress/e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx" Jul 23 00:48:59.311669 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:59.311465 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d216ac9-957c-46fa-9fa8-b612bda28053-istio-data\") pod \"e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx\" (UID: \"2d216ac9-957c-46fa-9fa8-b612bda28053\") " pod="openshift-ingress/e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx" Jul 23 00:48:59.311669 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:59.311488 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d216ac9-957c-46fa-9fa8-b612bda28053-istio-envoy\") pod \"e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx\" (UID: \"2d216ac9-957c-46fa-9fa8-b612bda28053\") " pod="openshift-ingress/e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx" Jul 23 00:48:59.311669 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:59.311521 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tkg6\" (UniqueName: \"kubernetes.io/projected/2d216ac9-957c-46fa-9fa8-b612bda28053-kube-api-access-5tkg6\") pod \"e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx\" (UID: \"2d216ac9-957c-46fa-9fa8-b612bda28053\") " pod="openshift-ingress/e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx" Jul 23 00:48:59.311669 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:59.311548 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d216ac9-957c-46fa-9fa8-b612bda28053-istio-podinfo\") pod \"e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx\" (UID: \"2d216ac9-957c-46fa-9fa8-b612bda28053\") " pod="openshift-ingress/e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx" Jul 23 00:48:59.311669 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:59.311583 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d216ac9-957c-46fa-9fa8-b612bda28053-workload-socket\") pod \"e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx\" (UID: \"2d216ac9-957c-46fa-9fa8-b612bda28053\") " pod="openshift-ingress/e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx" Jul 23 00:48:59.311669 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:59.311610 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d216ac9-957c-46fa-9fa8-b612bda28053-workload-certs\") pod \"e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx\" (UID: \"2d216ac9-957c-46fa-9fa8-b612bda28053\") " pod="openshift-ingress/e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx" Jul 23 00:48:59.312079 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:59.311962 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d216ac9-957c-46fa-9fa8-b612bda28053-credential-socket\") pod \"e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx\" (UID: \"2d216ac9-957c-46fa-9fa8-b612bda28053\") " pod="openshift-ingress/e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx" Jul 23 00:48:59.312079 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:59.312024 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d216ac9-957c-46fa-9fa8-b612bda28053-workload-certs\") pod \"e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx\" (UID: \"2d216ac9-957c-46fa-9fa8-b612bda28053\") " pod="openshift-ingress/e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx" Jul 23 00:48:59.312156 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:59.312102 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d216ac9-957c-46fa-9fa8-b612bda28053-workload-socket\") pod \"e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx\" (UID: \"2d216ac9-957c-46fa-9fa8-b612bda28053\") " pod="openshift-ingress/e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx" Jul 23 00:48:59.312191 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:59.312159 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d216ac9-957c-46fa-9fa8-b612bda28053-istio-data\") pod \"e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx\" (UID: \"2d216ac9-957c-46fa-9fa8-b612bda28053\") " pod="openshift-ingress/e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx" Jul 23 00:48:59.312318 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:59.312294 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d216ac9-957c-46fa-9fa8-b612bda28053-istiod-ca-cert\") pod \"e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx\" (UID: \"2d216ac9-957c-46fa-9fa8-b612bda28053\") " pod="openshift-ingress/e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx" Jul 23 00:48:59.314020 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:59.313996 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d216ac9-957c-46fa-9fa8-b612bda28053-istio-envoy\") pod \"e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx\" (UID: \"2d216ac9-957c-46fa-9fa8-b612bda28053\") " pod="openshift-ingress/e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx" Jul 23 00:48:59.314280 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:59.314256 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d216ac9-957c-46fa-9fa8-b612bda28053-istio-podinfo\") pod \"e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx\" (UID: \"2d216ac9-957c-46fa-9fa8-b612bda28053\") " pod="openshift-ingress/e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx" Jul 23 00:48:59.319599 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:59.319568 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d216ac9-957c-46fa-9fa8-b612bda28053-istio-token\") pod \"e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx\" (UID: \"2d216ac9-957c-46fa-9fa8-b612bda28053\") " pod="openshift-ingress/e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx" Jul 23 00:48:59.320148 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:59.320122 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tkg6\" (UniqueName: \"kubernetes.io/projected/2d216ac9-957c-46fa-9fa8-b612bda28053-kube-api-access-5tkg6\") pod \"e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx\" (UID: \"2d216ac9-957c-46fa-9fa8-b612bda28053\") " pod="openshift-ingress/e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx" Jul 23 00:48:59.481617 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:59.481509 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx" Jul 23 00:48:59.629596 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:59.629555 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx"] Jul 23 00:48:59.630788 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:48:59.630741 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d216ac9_957c_46fa_9fa8_b612bda28053.slice/crio-5d6175a1aff26d7aad64363fd5b30fc96d1e97a5ce29a0ead36a81778edcf86d WatchSource:0}: Error finding container 5d6175a1aff26d7aad64363fd5b30fc96d1e97a5ce29a0ead36a81778edcf86d: Status 404 returned error can't find the container with id 5d6175a1aff26d7aad64363fd5b30fc96d1e97a5ce29a0ead36a81778edcf86d Jul 23 00:48:59.633028 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:59.632994 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:48:59.633141 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:59.633075 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:48:59.633141 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:48:59.633119 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:49:00.597463 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:00.597426 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx" event={"ID":"2d216ac9-957c-46fa-9fa8-b612bda28053","Type":"ContainerStarted","Data":"d05dcd7e623f1e1735151cc2fa45bb1c76cc1296e26689e88faac2547a4b1cb3"} Jul 23 00:49:00.597893 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:00.597482 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx" event={"ID":"2d216ac9-957c-46fa-9fa8-b612bda28053","Type":"ContainerStarted","Data":"5d6175a1aff26d7aad64363fd5b30fc96d1e97a5ce29a0ead36a81778edcf86d"} Jul 23 00:49:00.620287 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:00.620220 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx" podStartSLOduration=1.620201766 podStartE2EDuration="1.620201766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:48:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:49:00.617522863 +0000 UTC m=+2454.413029722" watchObservedRunningTime="2026-07-23 00:49:00.620201766 +0000 UTC m=+2454.415708622" Jul 23 00:49:01.482294 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:01.482254 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx" Jul 23 00:49:01.487919 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:01.487889 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx" Jul 23 00:49:01.603371 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:01.603340 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx" Jul 23 00:49:01.604500 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:01.604477 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx" Jul 23 00:49:07.698672 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:07.698631 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt"] Jul 23 00:49:07.702054 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:07.702027 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt" Jul 23 00:49:07.704882 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:07.704859 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-a242b2-openshift-default-dockercfg-nk6gq\"" Jul 23 00:49:07.714405 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:07.714375 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt"] Jul 23 00:49:07.796797 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:07.796747 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4073de49-baee-4c6a-a68d-b4d52ecd9513-istio-data\") pod \"e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt\" (UID: \"4073de49-baee-4c6a-a68d-b4d52ecd9513\") " pod="openshift-ingress/e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt" Jul 23 00:49:07.796979 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:07.796803 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4073de49-baee-4c6a-a68d-b4d52ecd9513-istio-token\") pod \"e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt\" (UID: \"4073de49-baee-4c6a-a68d-b4d52ecd9513\") " pod="openshift-ingress/e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt" Jul 23 00:49:07.796979 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:07.796874 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4073de49-baee-4c6a-a68d-b4d52ecd9513-istio-podinfo\") pod \"e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt\" (UID: \"4073de49-baee-4c6a-a68d-b4d52ecd9513\") " pod="openshift-ingress/e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt" Jul 23 00:49:07.796979 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:07.796955 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4073de49-baee-4c6a-a68d-b4d52ecd9513-istio-envoy\") pod \"e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt\" (UID: \"4073de49-baee-4c6a-a68d-b4d52ecd9513\") " pod="openshift-ingress/e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt" Jul 23 00:49:07.797089 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:07.796999 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4073de49-baee-4c6a-a68d-b4d52ecd9513-workload-certs\") pod \"e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt\" (UID: \"4073de49-baee-4c6a-a68d-b4d52ecd9513\") " pod="openshift-ingress/e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt" Jul 23 00:49:07.797089 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:07.797021 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4073de49-baee-4c6a-a68d-b4d52ecd9513-credential-socket\") pod \"e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt\" (UID: \"4073de49-baee-4c6a-a68d-b4d52ecd9513\") " pod="openshift-ingress/e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt" Jul 23 00:49:07.797089 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:07.797042 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4073de49-baee-4c6a-a68d-b4d52ecd9513-workload-socket\") pod \"e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt\" (UID: \"4073de49-baee-4c6a-a68d-b4d52ecd9513\") " pod="openshift-ingress/e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt" Jul 23 00:49:07.797184 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:07.797115 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4073de49-baee-4c6a-a68d-b4d52ecd9513-istiod-ca-cert\") pod \"e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt\" (UID: \"4073de49-baee-4c6a-a68d-b4d52ecd9513\") " pod="openshift-ingress/e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt" Jul 23 00:49:07.797184 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:07.797173 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2ckx\" (UniqueName: \"kubernetes.io/projected/4073de49-baee-4c6a-a68d-b4d52ecd9513-kube-api-access-b2ckx\") pod \"e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt\" (UID: \"4073de49-baee-4c6a-a68d-b4d52ecd9513\") " pod="openshift-ingress/e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt" Jul 23 00:49:07.898120 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:07.898082 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4073de49-baee-4c6a-a68d-b4d52ecd9513-workload-certs\") pod \"e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt\" (UID: \"4073de49-baee-4c6a-a68d-b4d52ecd9513\") " pod="openshift-ingress/e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt" Jul 23 00:49:07.898120 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:07.898122 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4073de49-baee-4c6a-a68d-b4d52ecd9513-credential-socket\") pod \"e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt\" (UID: \"4073de49-baee-4c6a-a68d-b4d52ecd9513\") " pod="openshift-ingress/e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt" Jul 23 00:49:07.898357 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:07.898142 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4073de49-baee-4c6a-a68d-b4d52ecd9513-workload-socket\") pod \"e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt\" (UID: \"4073de49-baee-4c6a-a68d-b4d52ecd9513\") " pod="openshift-ingress/e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt" Jul 23 00:49:07.898357 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:07.898167 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4073de49-baee-4c6a-a68d-b4d52ecd9513-istiod-ca-cert\") pod \"e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt\" (UID: \"4073de49-baee-4c6a-a68d-b4d52ecd9513\") " pod="openshift-ingress/e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt" Jul 23 00:49:07.898357 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:07.898216 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2ckx\" (UniqueName: \"kubernetes.io/projected/4073de49-baee-4c6a-a68d-b4d52ecd9513-kube-api-access-b2ckx\") pod \"e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt\" (UID: \"4073de49-baee-4c6a-a68d-b4d52ecd9513\") " pod="openshift-ingress/e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt" Jul 23 00:49:07.898357 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:07.898261 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4073de49-baee-4c6a-a68d-b4d52ecd9513-istio-data\") pod \"e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt\" (UID: \"4073de49-baee-4c6a-a68d-b4d52ecd9513\") " pod="openshift-ingress/e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt" Jul 23 00:49:07.898357 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:07.898285 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4073de49-baee-4c6a-a68d-b4d52ecd9513-istio-token\") pod \"e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt\" (UID: \"4073de49-baee-4c6a-a68d-b4d52ecd9513\") " pod="openshift-ingress/e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt" Jul 23 00:49:07.898357 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:07.898336 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4073de49-baee-4c6a-a68d-b4d52ecd9513-istio-podinfo\") pod \"e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt\" (UID: \"4073de49-baee-4c6a-a68d-b4d52ecd9513\") " pod="openshift-ingress/e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt" Jul 23 00:49:07.898687 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:07.898377 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4073de49-baee-4c6a-a68d-b4d52ecd9513-istio-envoy\") pod \"e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt\" (UID: \"4073de49-baee-4c6a-a68d-b4d52ecd9513\") " pod="openshift-ingress/e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt" Jul 23 00:49:07.898687 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:07.898556 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4073de49-baee-4c6a-a68d-b4d52ecd9513-workload-certs\") pod \"e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt\" (UID: \"4073de49-baee-4c6a-a68d-b4d52ecd9513\") " pod="openshift-ingress/e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt" Jul 23 00:49:07.898687 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:07.898616 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4073de49-baee-4c6a-a68d-b4d52ecd9513-credential-socket\") pod \"e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt\" (UID: \"4073de49-baee-4c6a-a68d-b4d52ecd9513\") " pod="openshift-ingress/e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt" Jul 23 00:49:07.898942 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:07.898915 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4073de49-baee-4c6a-a68d-b4d52ecd9513-workload-socket\") pod \"e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt\" (UID: \"4073de49-baee-4c6a-a68d-b4d52ecd9513\") " pod="openshift-ingress/e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt" Jul 23 00:49:07.899149 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:07.899130 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4073de49-baee-4c6a-a68d-b4d52ecd9513-istio-data\") pod \"e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt\" (UID: \"4073de49-baee-4c6a-a68d-b4d52ecd9513\") " pod="openshift-ingress/e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt" Jul 23 00:49:07.899336 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:07.899305 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4073de49-baee-4c6a-a68d-b4d52ecd9513-istiod-ca-cert\") pod \"e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt\" (UID: \"4073de49-baee-4c6a-a68d-b4d52ecd9513\") " pod="openshift-ingress/e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt" Jul 23 00:49:07.900826 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:07.900802 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4073de49-baee-4c6a-a68d-b4d52ecd9513-istio-envoy\") pod \"e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt\" (UID: \"4073de49-baee-4c6a-a68d-b4d52ecd9513\") " pod="openshift-ingress/e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt" Jul 23 00:49:07.901389 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:07.901363 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4073de49-baee-4c6a-a68d-b4d52ecd9513-istio-podinfo\") pod \"e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt\" (UID: \"4073de49-baee-4c6a-a68d-b4d52ecd9513\") " pod="openshift-ingress/e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt" Jul 23 00:49:07.907035 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:07.907008 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4073de49-baee-4c6a-a68d-b4d52ecd9513-istio-token\") pod \"e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt\" (UID: \"4073de49-baee-4c6a-a68d-b4d52ecd9513\") " pod="openshift-ingress/e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt" Jul 23 00:49:07.907241 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:07.907217 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2ckx\" (UniqueName: \"kubernetes.io/projected/4073de49-baee-4c6a-a68d-b4d52ecd9513-kube-api-access-b2ckx\") pod \"e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt\" (UID: \"4073de49-baee-4c6a-a68d-b4d52ecd9513\") " pod="openshift-ingress/e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt" Jul 23 00:49:08.013825 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:08.013704 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt" Jul 23 00:49:08.164009 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:08.163978 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt"] Jul 23 00:49:08.166053 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:49:08.166025 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4073de49_baee_4c6a_a68d_b4d52ecd9513.slice/crio-ec16ef62a5251b1d117ce71b6391aebec181ce0b66441a23ffe8d5b3dc7913b9 WatchSource:0}: Error finding container ec16ef62a5251b1d117ce71b6391aebec181ce0b66441a23ffe8d5b3dc7913b9: Status 404 returned error can't find the container with id ec16ef62a5251b1d117ce71b6391aebec181ce0b66441a23ffe8d5b3dc7913b9 Jul 23 00:49:08.168899 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:08.168860 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:49:08.168983 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:08.168936 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:49:08.169023 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:08.168990 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:49:08.643629 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:08.643591 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt" event={"ID":"4073de49-baee-4c6a-a68d-b4d52ecd9513","Type":"ContainerStarted","Data":"a1b7679110d0116755d8aa56a47ee4a846b5adc519af8a18673a7dd99553dd53"} Jul 23 00:49:08.643816 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:08.643650 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt" event={"ID":"4073de49-baee-4c6a-a68d-b4d52ecd9513","Type":"ContainerStarted","Data":"ec16ef62a5251b1d117ce71b6391aebec181ce0b66441a23ffe8d5b3dc7913b9"} Jul 23 00:49:08.668496 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:08.668434 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt" podStartSLOduration=1.668414982 podStartE2EDuration="1.668414982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:49:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:49:08.666646243 +0000 UTC m=+2462.462153112" watchObservedRunningTime="2026-07-23 00:49:08.668414982 +0000 UTC m=+2462.463921839" Jul 23 00:49:09.014232 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:09.014147 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt" Jul 23 00:49:09.019402 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:09.019371 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt" Jul 23 00:49:09.648980 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:09.648941 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt" Jul 23 00:49:09.650001 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:09.649973 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt" Jul 23 00:49:16.604963 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:16.604878 2562 patch_prober.go:28] interesting pod/e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 23 00:49:16.605406 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:16.604966 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx" podUID="2d216ac9-957c-46fa-9fa8-b612bda28053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 23 00:49:16.690235 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:16.690149 2562 generic.go:356] "Generic (PLEG): container finished" podID="2d216ac9-957c-46fa-9fa8-b612bda28053" containerID="d05dcd7e623f1e1735151cc2fa45bb1c76cc1296e26689e88faac2547a4b1cb3" exitCode=0 Jul 23 00:49:16.690235 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:16.690222 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx" event={"ID":"2d216ac9-957c-46fa-9fa8-b612bda28053","Type":"ContainerDied","Data":"d05dcd7e623f1e1735151cc2fa45bb1c76cc1296e26689e88faac2547a4b1cb3"} Jul 23 00:49:16.690613 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:16.690599 2562 scope.go:117] "RemoveContainer" containerID="d05dcd7e623f1e1735151cc2fa45bb1c76cc1296e26689e88faac2547a4b1cb3" Jul 23 00:49:16.691564 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:16.691539 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:49:16.691620 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:16.691606 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:49:16.691667 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:16.691636 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:49:17.702192 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:17.702147 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx" event={"ID":"2d216ac9-957c-46fa-9fa8-b612bda28053","Type":"ContainerStarted","Data":"4b5f3d632598feda455e22102f05441b647f9c377532e8f0dac61ef05ae4b78d"} Jul 23 00:49:18.482284 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:18.482235 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx" Jul 23 00:49:18.482516 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:18.482486 2562 patch_prober.go:28] interesting pod/e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 23 00:49:18.482603 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:18.482552 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx" podUID="2d216ac9-957c-46fa-9fa8-b612bda28053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 23 00:49:19.482837 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:19.482798 2562 patch_prober.go:28] interesting pod/e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 23 00:49:19.483265 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:19.482861 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx" podUID="2d216ac9-957c-46fa-9fa8-b612bda28053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 23 00:49:20.482632 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:20.482583 2562 patch_prober.go:28] interesting pod/e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 23 00:49:20.482835 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:20.482646 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx" podUID="2d216ac9-957c-46fa-9fa8-b612bda28053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 23 00:49:21.482417 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:21.482374 2562 patch_prober.go:28] interesting pod/e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 23 00:49:21.482945 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:21.482459 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx" podUID="2d216ac9-957c-46fa-9fa8-b612bda28053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 23 00:49:22.486258 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:22.486220 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx" Jul 23 00:49:22.486783 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:22.486590 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx" Jul 23 00:49:22.487314 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:22.487288 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx" Jul 23 00:49:25.690950 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:25.690900 2562 patch_prober.go:28] interesting pod/e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 23 00:49:25.691396 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:25.690975 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt" podUID="4073de49-baee-4c6a-a68d-b4d52ecd9513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 23 00:49:27.772831 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:27.772793 2562 generic.go:356] "Generic (PLEG): container finished" podID="4073de49-baee-4c6a-a68d-b4d52ecd9513" containerID="a1b7679110d0116755d8aa56a47ee4a846b5adc519af8a18673a7dd99553dd53" exitCode=0 Jul 23 00:49:27.773401 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:27.772872 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt" event={"ID":"4073de49-baee-4c6a-a68d-b4d52ecd9513","Type":"ContainerDied","Data":"a1b7679110d0116755d8aa56a47ee4a846b5adc519af8a18673a7dd99553dd53"} Jul 23 00:49:27.773401 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:27.773342 2562 scope.go:117] "RemoveContainer" containerID="a1b7679110d0116755d8aa56a47ee4a846b5adc519af8a18673a7dd99553dd53" Jul 23 00:49:27.774390 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:27.774363 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:49:27.774444 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:27.774430 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:49:27.774480 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:27.774465 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:49:28.780474 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:28.780422 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt" event={"ID":"4073de49-baee-4c6a-a68d-b4d52ecd9513","Type":"ContainerStarted","Data":"f8e41c11e7ee1c39a3527ca5b0e8305e51795b214f2066594e1c914a3ed13dab"} Jul 23 00:49:29.014109 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:29.014066 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt" Jul 23 00:49:29.014317 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:29.014127 2562 patch_prober.go:28] interesting pod/e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 23 00:49:29.014317 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:29.014175 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt" podUID="4073de49-baee-4c6a-a68d-b4d52ecd9513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 23 00:49:30.014197 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:30.014147 2562 patch_prober.go:28] interesting pod/e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 23 00:49:30.014788 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:30.014222 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt" podUID="4073de49-baee-4c6a-a68d-b4d52ecd9513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 23 00:49:31.014891 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:31.014855 2562 patch_prober.go:28] interesting pod/e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 23 00:49:31.015618 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:31.015581 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt" podUID="4073de49-baee-4c6a-a68d-b4d52ecd9513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 23 00:49:32.014224 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:32.014171 2562 patch_prober.go:28] interesting pod/e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 23 00:49:32.014441 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:32.014249 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt" podUID="4073de49-baee-4c6a-a68d-b4d52ecd9513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 23 00:49:33.014515 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:33.014478 2562 patch_prober.go:28] interesting pod/e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 23 00:49:33.014948 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:33.014553 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt" podUID="4073de49-baee-4c6a-a68d-b4d52ecd9513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 23 00:49:34.018011 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:34.017978 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt" Jul 23 00:49:34.018416 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:34.018334 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt" Jul 23 00:49:34.019127 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:49:34.019108 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt" Jul 23 00:50:57.839741 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:50:57.839694 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx"] Jul 23 00:50:57.840856 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:50:57.840747 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx" podUID="2d216ac9-957c-46fa-9fa8-b612bda28053" containerName="istio-proxy" containerID="cri-o://4b5f3d632598feda455e22102f05441b647f9c377532e8f0dac61ef05ae4b78d" gracePeriod=30 Jul 23 00:51:03.099508 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:03.099434 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx" Jul 23 00:51:03.243376 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:03.243337 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d216ac9-957c-46fa-9fa8-b612bda28053-istio-data\") pod \"2d216ac9-957c-46fa-9fa8-b612bda28053\" (UID: \"2d216ac9-957c-46fa-9fa8-b612bda28053\") " Jul 23 00:51:03.243557 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:03.243403 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5tkg6\" (UniqueName: \"kubernetes.io/projected/2d216ac9-957c-46fa-9fa8-b612bda28053-kube-api-access-5tkg6\") pod \"2d216ac9-957c-46fa-9fa8-b612bda28053\" (UID: \"2d216ac9-957c-46fa-9fa8-b612bda28053\") " Jul 23 00:51:03.243557 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:03.243430 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d216ac9-957c-46fa-9fa8-b612bda28053-credential-socket\") pod \"2d216ac9-957c-46fa-9fa8-b612bda28053\" (UID: \"2d216ac9-957c-46fa-9fa8-b612bda28053\") " Jul 23 00:51:03.243557 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:03.243472 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d216ac9-957c-46fa-9fa8-b612bda28053-istio-envoy\") pod \"2d216ac9-957c-46fa-9fa8-b612bda28053\" (UID: \"2d216ac9-957c-46fa-9fa8-b612bda28053\") " Jul 23 00:51:03.243557 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:03.243528 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d216ac9-957c-46fa-9fa8-b612bda28053-istiod-ca-cert\") pod \"2d216ac9-957c-46fa-9fa8-b612bda28053\" (UID: \"2d216ac9-957c-46fa-9fa8-b612bda28053\") " Jul 23 00:51:03.243743 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:03.243562 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d216ac9-957c-46fa-9fa8-b612bda28053-workload-socket\") pod \"2d216ac9-957c-46fa-9fa8-b612bda28053\" (UID: \"2d216ac9-957c-46fa-9fa8-b612bda28053\") " Jul 23 00:51:03.243743 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:03.243595 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d216ac9-957c-46fa-9fa8-b612bda28053-istio-token\") pod \"2d216ac9-957c-46fa-9fa8-b612bda28053\" (UID: \"2d216ac9-957c-46fa-9fa8-b612bda28053\") " Jul 23 00:51:03.243743 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:03.243641 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d216ac9-957c-46fa-9fa8-b612bda28053-istio-podinfo\") pod \"2d216ac9-957c-46fa-9fa8-b612bda28053\" (UID: \"2d216ac9-957c-46fa-9fa8-b612bda28053\") " Jul 23 00:51:03.243743 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:03.243667 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d216ac9-957c-46fa-9fa8-b612bda28053-workload-certs\") pod \"2d216ac9-957c-46fa-9fa8-b612bda28053\" (UID: \"2d216ac9-957c-46fa-9fa8-b612bda28053\") " Jul 23 00:51:03.243987 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:03.243801 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d216ac9-957c-46fa-9fa8-b612bda28053-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2d216ac9-957c-46fa-9fa8-b612bda28053" (UID: "2d216ac9-957c-46fa-9fa8-b612bda28053"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:51:03.243987 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:03.243862 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d216ac9-957c-46fa-9fa8-b612bda28053-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2d216ac9-957c-46fa-9fa8-b612bda28053" (UID: "2d216ac9-957c-46fa-9fa8-b612bda28053"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:51:03.244096 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:03.243988 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d216ac9-957c-46fa-9fa8-b612bda28053-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2d216ac9-957c-46fa-9fa8-b612bda28053" (UID: "2d216ac9-957c-46fa-9fa8-b612bda28053"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:51:03.244152 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:03.244092 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d216ac9-957c-46fa-9fa8-b612bda28053-credential-socket\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:51:03.244152 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:03.244114 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d216ac9-957c-46fa-9fa8-b612bda28053-istiod-ca-cert\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:51:03.244152 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:03.244130 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d216ac9-957c-46fa-9fa8-b612bda28053-workload-socket\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:51:03.244389 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:03.244323 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d216ac9-957c-46fa-9fa8-b612bda28053-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2d216ac9-957c-46fa-9fa8-b612bda28053" (UID: "2d216ac9-957c-46fa-9fa8-b612bda28053"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:51:03.244478 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:03.244452 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d216ac9-957c-46fa-9fa8-b612bda28053-istio-data" (OuterVolumeSpecName: "istio-data") pod "2d216ac9-957c-46fa-9fa8-b612bda28053" (UID: "2d216ac9-957c-46fa-9fa8-b612bda28053"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:51:03.245878 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:03.245844 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d216ac9-957c-46fa-9fa8-b612bda28053-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2d216ac9-957c-46fa-9fa8-b612bda28053" (UID: "2d216ac9-957c-46fa-9fa8-b612bda28053"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:51:03.245878 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:03.245872 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d216ac9-957c-46fa-9fa8-b612bda28053-istio-token" (OuterVolumeSpecName: "istio-token") pod "2d216ac9-957c-46fa-9fa8-b612bda28053" (UID: "2d216ac9-957c-46fa-9fa8-b612bda28053"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:51:03.246026 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:03.245884 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d216ac9-957c-46fa-9fa8-b612bda28053-kube-api-access-5tkg6" (OuterVolumeSpecName: "kube-api-access-5tkg6") pod "2d216ac9-957c-46fa-9fa8-b612bda28053" (UID: "2d216ac9-957c-46fa-9fa8-b612bda28053"). InnerVolumeSpecName "kube-api-access-5tkg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:51:03.246026 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:03.245926 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d216ac9-957c-46fa-9fa8-b612bda28053-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2d216ac9-957c-46fa-9fa8-b612bda28053" (UID: "2d216ac9-957c-46fa-9fa8-b612bda28053"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 00:51:03.335059 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:03.335020 2562 generic.go:356] "Generic (PLEG): container finished" podID="2d216ac9-957c-46fa-9fa8-b612bda28053" containerID="4b5f3d632598feda455e22102f05441b647f9c377532e8f0dac61ef05ae4b78d" exitCode=0 Jul 23 00:51:03.335236 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:03.335068 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx" Jul 23 00:51:03.335236 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:03.335100 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx" event={"ID":"2d216ac9-957c-46fa-9fa8-b612bda28053","Type":"ContainerDied","Data":"4b5f3d632598feda455e22102f05441b647f9c377532e8f0dac61ef05ae4b78d"} Jul 23 00:51:03.335236 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:03.335161 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx" event={"ID":"2d216ac9-957c-46fa-9fa8-b612bda28053","Type":"ContainerDied","Data":"5d6175a1aff26d7aad64363fd5b30fc96d1e97a5ce29a0ead36a81778edcf86d"} Jul 23 00:51:03.335236 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:03.335192 2562 scope.go:117] "RemoveContainer" containerID="4b5f3d632598feda455e22102f05441b647f9c377532e8f0dac61ef05ae4b78d" Jul 23 00:51:03.345074 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:03.345047 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d216ac9-957c-46fa-9fa8-b612bda28053-istio-data\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:51:03.345199 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:03.345074 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tkg6\" (UniqueName: \"kubernetes.io/projected/2d216ac9-957c-46fa-9fa8-b612bda28053-kube-api-access-5tkg6\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:51:03.345199 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:03.345090 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d216ac9-957c-46fa-9fa8-b612bda28053-istio-envoy\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:51:03.345199 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:03.345106 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d216ac9-957c-46fa-9fa8-b612bda28053-istio-token\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:51:03.345199 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:03.345119 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d216ac9-957c-46fa-9fa8-b612bda28053-istio-podinfo\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:51:03.345199 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:03.345130 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d216ac9-957c-46fa-9fa8-b612bda28053-workload-certs\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:51:03.345593 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:03.345577 2562 scope.go:117] "RemoveContainer" containerID="d05dcd7e623f1e1735151cc2fa45bb1c76cc1296e26689e88faac2547a4b1cb3" Jul 23 00:51:03.354747 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:03.354727 2562 scope.go:117] "RemoveContainer" containerID="4b5f3d632598feda455e22102f05441b647f9c377532e8f0dac61ef05ae4b78d" Jul 23 00:51:03.355080 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:51:03.355060 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b5f3d632598feda455e22102f05441b647f9c377532e8f0dac61ef05ae4b78d\": container with ID starting with 4b5f3d632598feda455e22102f05441b647f9c377532e8f0dac61ef05ae4b78d not found: ID does not exist" containerID="4b5f3d632598feda455e22102f05441b647f9c377532e8f0dac61ef05ae4b78d" Jul 23 00:51:03.355128 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:03.355090 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b5f3d632598feda455e22102f05441b647f9c377532e8f0dac61ef05ae4b78d"} err="failed to get container status \"4b5f3d632598feda455e22102f05441b647f9c377532e8f0dac61ef05ae4b78d\": rpc error: code = NotFound desc = could not find container \"4b5f3d632598feda455e22102f05441b647f9c377532e8f0dac61ef05ae4b78d\": container with ID starting with 4b5f3d632598feda455e22102f05441b647f9c377532e8f0dac61ef05ae4b78d not found: ID does not exist" Jul 23 00:51:03.355128 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:03.355109 2562 scope.go:117] "RemoveContainer" containerID="d05dcd7e623f1e1735151cc2fa45bb1c76cc1296e26689e88faac2547a4b1cb3" Jul 23 00:51:03.355350 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:51:03.355332 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d05dcd7e623f1e1735151cc2fa45bb1c76cc1296e26689e88faac2547a4b1cb3\": container with ID starting with d05dcd7e623f1e1735151cc2fa45bb1c76cc1296e26689e88faac2547a4b1cb3 not found: ID does not exist" containerID="d05dcd7e623f1e1735151cc2fa45bb1c76cc1296e26689e88faac2547a4b1cb3" Jul 23 00:51:03.355391 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:03.355358 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d05dcd7e623f1e1735151cc2fa45bb1c76cc1296e26689e88faac2547a4b1cb3"} err="failed to get container status \"d05dcd7e623f1e1735151cc2fa45bb1c76cc1296e26689e88faac2547a4b1cb3\": rpc error: code = NotFound desc = could not find container \"d05dcd7e623f1e1735151cc2fa45bb1c76cc1296e26689e88faac2547a4b1cb3\": container with ID starting with d05dcd7e623f1e1735151cc2fa45bb1c76cc1296e26689e88faac2547a4b1cb3 not found: ID does not exist" Jul 23 00:51:03.359553 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:03.359520 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx"] Jul 23 00:51:03.365260 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:03.365227 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-e5bc11-openshift-default-84cf957ff9-f7pqx"] Jul 23 00:51:05.007391 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:05.007353 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d216ac9-957c-46fa-9fa8-b612bda28053" path="/var/lib/kubelet/pods/2d216ac9-957c-46fa-9fa8-b612bda28053/volumes" Jul 23 00:51:19.781242 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:19.781204 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt"] Jul 23 00:51:19.781653 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:19.781422 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt" podUID="4073de49-baee-4c6a-a68d-b4d52ecd9513" containerName="istio-proxy" containerID="cri-o://f8e41c11e7ee1c39a3527ca5b0e8305e51795b214f2066594e1c914a3ed13dab" gracePeriod=30 Jul 23 00:51:25.032375 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:25.032347 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt" Jul 23 00:51:25.043361 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:25.043289 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4073de49-baee-4c6a-a68d-b4d52ecd9513-credential-socket\") pod \"4073de49-baee-4c6a-a68d-b4d52ecd9513\" (UID: \"4073de49-baee-4c6a-a68d-b4d52ecd9513\") " Jul 23 00:51:25.043361 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:25.043324 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4073de49-baee-4c6a-a68d-b4d52ecd9513-istio-data\") pod \"4073de49-baee-4c6a-a68d-b4d52ecd9513\" (UID: \"4073de49-baee-4c6a-a68d-b4d52ecd9513\") " Jul 23 00:51:25.043361 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:25.043360 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b2ckx\" (UniqueName: \"kubernetes.io/projected/4073de49-baee-4c6a-a68d-b4d52ecd9513-kube-api-access-b2ckx\") pod \"4073de49-baee-4c6a-a68d-b4d52ecd9513\" (UID: \"4073de49-baee-4c6a-a68d-b4d52ecd9513\") " Jul 23 00:51:25.043597 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:25.043386 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4073de49-baee-4c6a-a68d-b4d52ecd9513-workload-socket\") pod \"4073de49-baee-4c6a-a68d-b4d52ecd9513\" (UID: \"4073de49-baee-4c6a-a68d-b4d52ecd9513\") " Jul 23 00:51:25.043597 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:25.043413 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4073de49-baee-4c6a-a68d-b4d52ecd9513-workload-certs\") pod \"4073de49-baee-4c6a-a68d-b4d52ecd9513\" (UID: \"4073de49-baee-4c6a-a68d-b4d52ecd9513\") " Jul 23 00:51:25.043597 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:25.043442 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4073de49-baee-4c6a-a68d-b4d52ecd9513-istio-token\") pod \"4073de49-baee-4c6a-a68d-b4d52ecd9513\" (UID: \"4073de49-baee-4c6a-a68d-b4d52ecd9513\") " Jul 23 00:51:25.043597 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:25.043478 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4073de49-baee-4c6a-a68d-b4d52ecd9513-istio-podinfo\") pod \"4073de49-baee-4c6a-a68d-b4d52ecd9513\" (UID: \"4073de49-baee-4c6a-a68d-b4d52ecd9513\") " Jul 23 00:51:25.043597 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:25.043530 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4073de49-baee-4c6a-a68d-b4d52ecd9513-istio-envoy\") pod \"4073de49-baee-4c6a-a68d-b4d52ecd9513\" (UID: \"4073de49-baee-4c6a-a68d-b4d52ecd9513\") " Jul 23 00:51:25.043597 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:25.043559 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4073de49-baee-4c6a-a68d-b4d52ecd9513-istiod-ca-cert\") pod \"4073de49-baee-4c6a-a68d-b4d52ecd9513\" (UID: \"4073de49-baee-4c6a-a68d-b4d52ecd9513\") " Jul 23 00:51:25.043597 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:25.043558 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4073de49-baee-4c6a-a68d-b4d52ecd9513-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4073de49-baee-4c6a-a68d-b4d52ecd9513" (UID: "4073de49-baee-4c6a-a68d-b4d52ecd9513"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:51:25.043977 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:25.043748 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4073de49-baee-4c6a-a68d-b4d52ecd9513-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4073de49-baee-4c6a-a68d-b4d52ecd9513" (UID: "4073de49-baee-4c6a-a68d-b4d52ecd9513"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:51:25.043977 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:25.043869 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4073de49-baee-4c6a-a68d-b4d52ecd9513-workload-socket\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:51:25.043977 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:25.043890 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4073de49-baee-4c6a-a68d-b4d52ecd9513-credential-socket\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:51:25.043977 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:25.043962 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4073de49-baee-4c6a-a68d-b4d52ecd9513-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4073de49-baee-4c6a-a68d-b4d52ecd9513" (UID: "4073de49-baee-4c6a-a68d-b4d52ecd9513"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:51:25.044209 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:25.044053 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4073de49-baee-4c6a-a68d-b4d52ecd9513-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4073de49-baee-4c6a-a68d-b4d52ecd9513" (UID: "4073de49-baee-4c6a-a68d-b4d52ecd9513"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:51:25.044419 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:25.044392 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4073de49-baee-4c6a-a68d-b4d52ecd9513-istio-data" (OuterVolumeSpecName: "istio-data") pod "4073de49-baee-4c6a-a68d-b4d52ecd9513" (UID: "4073de49-baee-4c6a-a68d-b4d52ecd9513"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:51:25.046183 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:25.046153 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4073de49-baee-4c6a-a68d-b4d52ecd9513-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4073de49-baee-4c6a-a68d-b4d52ecd9513" (UID: "4073de49-baee-4c6a-a68d-b4d52ecd9513"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:51:25.046274 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:25.046188 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4073de49-baee-4c6a-a68d-b4d52ecd9513-istio-token" (OuterVolumeSpecName: "istio-token") pod "4073de49-baee-4c6a-a68d-b4d52ecd9513" (UID: "4073de49-baee-4c6a-a68d-b4d52ecd9513"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:51:25.046274 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:25.046249 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4073de49-baee-4c6a-a68d-b4d52ecd9513-kube-api-access-b2ckx" (OuterVolumeSpecName: "kube-api-access-b2ckx") pod "4073de49-baee-4c6a-a68d-b4d52ecd9513" (UID: "4073de49-baee-4c6a-a68d-b4d52ecd9513"). InnerVolumeSpecName "kube-api-access-b2ckx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:51:25.046354 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:25.046321 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4073de49-baee-4c6a-a68d-b4d52ecd9513-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4073de49-baee-4c6a-a68d-b4d52ecd9513" (UID: "4073de49-baee-4c6a-a68d-b4d52ecd9513"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 00:51:25.145123 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:25.145083 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4073de49-baee-4c6a-a68d-b4d52ecd9513-istio-envoy\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:51:25.145123 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:25.145117 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4073de49-baee-4c6a-a68d-b4d52ecd9513-istiod-ca-cert\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:51:25.145123 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:25.145125 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4073de49-baee-4c6a-a68d-b4d52ecd9513-istio-data\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:51:25.145123 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:25.145133 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2ckx\" (UniqueName: \"kubernetes.io/projected/4073de49-baee-4c6a-a68d-b4d52ecd9513-kube-api-access-b2ckx\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:51:25.145408 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:25.145143 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4073de49-baee-4c6a-a68d-b4d52ecd9513-workload-certs\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:51:25.145408 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:25.145152 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4073de49-baee-4c6a-a68d-b4d52ecd9513-istio-token\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:51:25.145408 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:25.145159 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4073de49-baee-4c6a-a68d-b4d52ecd9513-istio-podinfo\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:51:25.460594 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:25.460495 2562 generic.go:356] "Generic (PLEG): container finished" podID="4073de49-baee-4c6a-a68d-b4d52ecd9513" containerID="f8e41c11e7ee1c39a3527ca5b0e8305e51795b214f2066594e1c914a3ed13dab" exitCode=0 Jul 23 00:51:25.460594 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:25.460546 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt" Jul 23 00:51:25.460594 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:25.460578 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt" event={"ID":"4073de49-baee-4c6a-a68d-b4d52ecd9513","Type":"ContainerDied","Data":"f8e41c11e7ee1c39a3527ca5b0e8305e51795b214f2066594e1c914a3ed13dab"} Jul 23 00:51:25.460901 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:25.460635 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt" event={"ID":"4073de49-baee-4c6a-a68d-b4d52ecd9513","Type":"ContainerDied","Data":"ec16ef62a5251b1d117ce71b6391aebec181ce0b66441a23ffe8d5b3dc7913b9"} Jul 23 00:51:25.460901 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:25.460661 2562 scope.go:117] "RemoveContainer" containerID="f8e41c11e7ee1c39a3527ca5b0e8305e51795b214f2066594e1c914a3ed13dab" Jul 23 00:51:25.470595 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:25.470574 2562 scope.go:117] "RemoveContainer" containerID="a1b7679110d0116755d8aa56a47ee4a846b5adc519af8a18673a7dd99553dd53" Jul 23 00:51:25.479497 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:25.479475 2562 scope.go:117] "RemoveContainer" containerID="f8e41c11e7ee1c39a3527ca5b0e8305e51795b214f2066594e1c914a3ed13dab" Jul 23 00:51:25.479849 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:51:25.479827 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8e41c11e7ee1c39a3527ca5b0e8305e51795b214f2066594e1c914a3ed13dab\": container with ID starting with f8e41c11e7ee1c39a3527ca5b0e8305e51795b214f2066594e1c914a3ed13dab not found: ID does not exist" containerID="f8e41c11e7ee1c39a3527ca5b0e8305e51795b214f2066594e1c914a3ed13dab" Jul 23 00:51:25.479910 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:25.479861 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8e41c11e7ee1c39a3527ca5b0e8305e51795b214f2066594e1c914a3ed13dab"} err="failed to get container status \"f8e41c11e7ee1c39a3527ca5b0e8305e51795b214f2066594e1c914a3ed13dab\": rpc error: code = NotFound desc = could not find container \"f8e41c11e7ee1c39a3527ca5b0e8305e51795b214f2066594e1c914a3ed13dab\": container with ID starting with f8e41c11e7ee1c39a3527ca5b0e8305e51795b214f2066594e1c914a3ed13dab not found: ID does not exist" Jul 23 00:51:25.479910 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:25.479882 2562 scope.go:117] "RemoveContainer" containerID="a1b7679110d0116755d8aa56a47ee4a846b5adc519af8a18673a7dd99553dd53" Jul 23 00:51:25.480141 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:51:25.480123 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1b7679110d0116755d8aa56a47ee4a846b5adc519af8a18673a7dd99553dd53\": container with ID starting with a1b7679110d0116755d8aa56a47ee4a846b5adc519af8a18673a7dd99553dd53 not found: ID does not exist" containerID="a1b7679110d0116755d8aa56a47ee4a846b5adc519af8a18673a7dd99553dd53" Jul 23 00:51:25.480185 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:25.480148 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1b7679110d0116755d8aa56a47ee4a846b5adc519af8a18673a7dd99553dd53"} err="failed to get container status \"a1b7679110d0116755d8aa56a47ee4a846b5adc519af8a18673a7dd99553dd53\": rpc error: code = NotFound desc = could not find container \"a1b7679110d0116755d8aa56a47ee4a846b5adc519af8a18673a7dd99553dd53\": container with ID starting with a1b7679110d0116755d8aa56a47ee4a846b5adc519af8a18673a7dd99553dd53 not found: ID does not exist" Jul 23 00:51:25.484089 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:25.484059 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt"] Jul 23 00:51:25.489513 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:25.489484 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-a242b2-openshift-default-774b4697b9-h24mt"] Jul 23 00:51:27.008014 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:27.007975 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4073de49-baee-4c6a-a68d-b4d52ecd9513" path="/var/lib/kubelet/pods/4073de49-baee-4c6a-a68d-b4d52ecd9513/volumes" Jul 23 00:51:28.726140 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:28.726055 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd"] Jul 23 00:51:28.726596 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:28.726458 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4073de49-baee-4c6a-a68d-b4d52ecd9513" containerName="istio-proxy" Jul 23 00:51:28.726596 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:28.726473 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4073de49-baee-4c6a-a68d-b4d52ecd9513" containerName="istio-proxy" Jul 23 00:51:28.726596 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:28.726484 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d216ac9-957c-46fa-9fa8-b612bda28053" containerName="istio-proxy" Jul 23 00:51:28.726596 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:28.726489 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d216ac9-957c-46fa-9fa8-b612bda28053" containerName="istio-proxy" Jul 23 00:51:28.726596 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:28.726497 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4073de49-baee-4c6a-a68d-b4d52ecd9513" containerName="istio-proxy" Jul 23 00:51:28.726596 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:28.726504 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4073de49-baee-4c6a-a68d-b4d52ecd9513" containerName="istio-proxy" Jul 23 00:51:28.726596 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:28.726591 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d216ac9-957c-46fa-9fa8-b612bda28053" containerName="istio-proxy" Jul 23 00:51:28.726596 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:28.726600 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="4073de49-baee-4c6a-a68d-b4d52ecd9513" containerName="istio-proxy" Jul 23 00:51:28.726903 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:28.726607 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="4073de49-baee-4c6a-a68d-b4d52ecd9513" containerName="istio-proxy" Jul 23 00:51:28.731277 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:28.731248 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd" Jul 23 00:51:28.734298 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:28.734275 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-e2debd-openshift-default-dockercfg-grzvq\"" Jul 23 00:51:28.744132 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:28.744111 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd"] Jul 23 00:51:28.775592 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:28.775554 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57c593fd-2b9c-4640-a49f-e70809c6e461-credential-socket\") pod \"e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd\" (UID: \"57c593fd-2b9c-4640-a49f-e70809c6e461\") " pod="openshift-ingress/e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd" Jul 23 00:51:28.775592 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:28.775606 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57c593fd-2b9c-4640-a49f-e70809c6e461-workload-socket\") pod \"e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd\" (UID: \"57c593fd-2b9c-4640-a49f-e70809c6e461\") " pod="openshift-ingress/e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd" Jul 23 00:51:28.775917 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:28.775637 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57c593fd-2b9c-4640-a49f-e70809c6e461-istio-token\") pod \"e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd\" (UID: \"57c593fd-2b9c-4640-a49f-e70809c6e461\") " pod="openshift-ingress/e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd" Jul 23 00:51:28.775917 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:28.775739 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57c593fd-2b9c-4640-a49f-e70809c6e461-istio-podinfo\") pod \"e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd\" (UID: \"57c593fd-2b9c-4640-a49f-e70809c6e461\") " pod="openshift-ingress/e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd" Jul 23 00:51:28.775917 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:28.775837 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57c593fd-2b9c-4640-a49f-e70809c6e461-workload-certs\") pod \"e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd\" (UID: \"57c593fd-2b9c-4640-a49f-e70809c6e461\") " pod="openshift-ingress/e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd" Jul 23 00:51:28.775917 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:28.775870 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57c593fd-2b9c-4640-a49f-e70809c6e461-istio-envoy\") pod \"e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd\" (UID: \"57c593fd-2b9c-4640-a49f-e70809c6e461\") " pod="openshift-ingress/e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd" Jul 23 00:51:28.775917 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:28.775893 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57c593fd-2b9c-4640-a49f-e70809c6e461-istio-data\") pod \"e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd\" (UID: \"57c593fd-2b9c-4640-a49f-e70809c6e461\") " pod="openshift-ingress/e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd" Jul 23 00:51:28.776158 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:28.775953 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57c593fd-2b9c-4640-a49f-e70809c6e461-istiod-ca-cert\") pod \"e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd\" (UID: \"57c593fd-2b9c-4640-a49f-e70809c6e461\") " pod="openshift-ingress/e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd" Jul 23 00:51:28.776158 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:28.775970 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dggb\" (UniqueName: \"kubernetes.io/projected/57c593fd-2b9c-4640-a49f-e70809c6e461-kube-api-access-4dggb\") pod \"e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd\" (UID: \"57c593fd-2b9c-4640-a49f-e70809c6e461\") " pod="openshift-ingress/e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd" Jul 23 00:51:28.877336 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:28.877296 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57c593fd-2b9c-4640-a49f-e70809c6e461-istio-podinfo\") pod \"e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd\" (UID: \"57c593fd-2b9c-4640-a49f-e70809c6e461\") " pod="openshift-ingress/e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd" Jul 23 00:51:28.877336 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:28.877339 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57c593fd-2b9c-4640-a49f-e70809c6e461-workload-certs\") pod \"e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd\" (UID: \"57c593fd-2b9c-4640-a49f-e70809c6e461\") " pod="openshift-ingress/e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd" Jul 23 00:51:28.877580 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:28.877385 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57c593fd-2b9c-4640-a49f-e70809c6e461-istio-envoy\") pod \"e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd\" (UID: \"57c593fd-2b9c-4640-a49f-e70809c6e461\") " pod="openshift-ingress/e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd" Jul 23 00:51:28.877580 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:28.877419 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57c593fd-2b9c-4640-a49f-e70809c6e461-istio-data\") pod \"e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd\" (UID: \"57c593fd-2b9c-4640-a49f-e70809c6e461\") " pod="openshift-ingress/e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd" Jul 23 00:51:28.877580 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:28.877486 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57c593fd-2b9c-4640-a49f-e70809c6e461-istiod-ca-cert\") pod \"e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd\" (UID: \"57c593fd-2b9c-4640-a49f-e70809c6e461\") " pod="openshift-ingress/e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd" Jul 23 00:51:28.877580 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:28.877511 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dggb\" (UniqueName: \"kubernetes.io/projected/57c593fd-2b9c-4640-a49f-e70809c6e461-kube-api-access-4dggb\") pod \"e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd\" (UID: \"57c593fd-2b9c-4640-a49f-e70809c6e461\") " pod="openshift-ingress/e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd" Jul 23 00:51:28.877580 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:28.877577 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57c593fd-2b9c-4640-a49f-e70809c6e461-credential-socket\") pod \"e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd\" (UID: \"57c593fd-2b9c-4640-a49f-e70809c6e461\") " pod="openshift-ingress/e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd" Jul 23 00:51:28.877872 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:28.877632 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57c593fd-2b9c-4640-a49f-e70809c6e461-workload-socket\") pod \"e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd\" (UID: \"57c593fd-2b9c-4640-a49f-e70809c6e461\") " pod="openshift-ingress/e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd" Jul 23 00:51:28.877872 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:28.877663 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57c593fd-2b9c-4640-a49f-e70809c6e461-istio-token\") pod \"e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd\" (UID: \"57c593fd-2b9c-4640-a49f-e70809c6e461\") " pod="openshift-ingress/e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd" Jul 23 00:51:28.877872 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:28.877691 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57c593fd-2b9c-4640-a49f-e70809c6e461-workload-certs\") pod \"e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd\" (UID: \"57c593fd-2b9c-4640-a49f-e70809c6e461\") " pod="openshift-ingress/e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd" Jul 23 00:51:28.878041 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:28.877998 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57c593fd-2b9c-4640-a49f-e70809c6e461-credential-socket\") pod \"e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd\" (UID: \"57c593fd-2b9c-4640-a49f-e70809c6e461\") " pod="openshift-ingress/e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd" Jul 23 00:51:28.878091 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:28.878070 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57c593fd-2b9c-4640-a49f-e70809c6e461-workload-socket\") pod \"e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd\" (UID: \"57c593fd-2b9c-4640-a49f-e70809c6e461\") " pod="openshift-ingress/e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd" Jul 23 00:51:28.878419 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:28.878392 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57c593fd-2b9c-4640-a49f-e70809c6e461-istiod-ca-cert\") pod \"e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd\" (UID: \"57c593fd-2b9c-4640-a49f-e70809c6e461\") " pod="openshift-ingress/e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd" Jul 23 00:51:28.878553 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:28.878533 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57c593fd-2b9c-4640-a49f-e70809c6e461-istio-data\") pod \"e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd\" (UID: \"57c593fd-2b9c-4640-a49f-e70809c6e461\") " pod="openshift-ingress/e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd" Jul 23 00:51:28.879653 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:28.879628 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57c593fd-2b9c-4640-a49f-e70809c6e461-istio-envoy\") pod \"e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd\" (UID: \"57c593fd-2b9c-4640-a49f-e70809c6e461\") " pod="openshift-ingress/e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd" Jul 23 00:51:28.880003 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:28.879985 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57c593fd-2b9c-4640-a49f-e70809c6e461-istio-podinfo\") pod \"e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd\" (UID: \"57c593fd-2b9c-4640-a49f-e70809c6e461\") " pod="openshift-ingress/e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd" Jul 23 00:51:28.886196 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:28.886174 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57c593fd-2b9c-4640-a49f-e70809c6e461-istio-token\") pod \"e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd\" (UID: \"57c593fd-2b9c-4640-a49f-e70809c6e461\") " pod="openshift-ingress/e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd" Jul 23 00:51:28.886368 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:28.886351 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dggb\" (UniqueName: \"kubernetes.io/projected/57c593fd-2b9c-4640-a49f-e70809c6e461-kube-api-access-4dggb\") pod \"e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd\" (UID: \"57c593fd-2b9c-4640-a49f-e70809c6e461\") " pod="openshift-ingress/e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd" Jul 23 00:51:29.042978 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:29.042938 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd" Jul 23 00:51:29.189298 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:29.189268 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd"] Jul 23 00:51:29.191042 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:51:29.191009 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57c593fd_2b9c_4640_a49f_e70809c6e461.slice/crio-ee96c98f0898df316df56ff0e1f973be6efb1c76ec3b39dddffeaeb2eabec5b1 WatchSource:0}: Error finding container ee96c98f0898df316df56ff0e1f973be6efb1c76ec3b39dddffeaeb2eabec5b1: Status 404 returned error can't find the container with id ee96c98f0898df316df56ff0e1f973be6efb1c76ec3b39dddffeaeb2eabec5b1 Jul 23 00:51:29.193427 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:29.193396 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:51:29.193496 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:29.193463 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:51:29.193541 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:29.193494 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:51:29.485845 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:29.485732 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd" event={"ID":"57c593fd-2b9c-4640-a49f-e70809c6e461","Type":"ContainerStarted","Data":"0724f19a60dd45ab1c78e47b7da3abd426feaab05ec477cf3f2f40be0c121722"} Jul 23 00:51:29.485845 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:29.485814 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd" event={"ID":"57c593fd-2b9c-4640-a49f-e70809c6e461","Type":"ContainerStarted","Data":"ee96c98f0898df316df56ff0e1f973be6efb1c76ec3b39dddffeaeb2eabec5b1"} Jul 23 00:51:29.507119 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:29.507066 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd" podStartSLOduration=1.507044738 podStartE2EDuration="1.507044738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:51:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:51:29.505810362 +0000 UTC m=+2603.301317220" watchObservedRunningTime="2026-07-23 00:51:29.507044738 +0000 UTC m=+2603.302551598" Jul 23 00:51:30.043866 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:30.043828 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd" Jul 23 00:51:30.049061 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:30.049032 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd" Jul 23 00:51:30.492099 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:30.492005 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd" Jul 23 00:51:30.493145 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:30.493125 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd" Jul 23 00:51:44.130449 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:44.130407 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-16466f-5pgqc"] Jul 23 00:51:44.130886 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:44.130848 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d216ac9-957c-46fa-9fa8-b612bda28053" containerName="istio-proxy" Jul 23 00:51:44.130886 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:44.130866 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d216ac9-957c-46fa-9fa8-b612bda28053" containerName="istio-proxy" Jul 23 00:51:44.130973 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:44.130961 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d216ac9-957c-46fa-9fa8-b612bda28053" containerName="istio-proxy" Jul 23 00:51:44.134087 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:44.134066 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-16466f-5pgqc" Jul 23 00:51:44.137108 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:44.137075 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w8tk2\"" Jul 23 00:51:44.142476 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:44.142448 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-16466f-5pgqc"] Jul 23 00:51:44.240217 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:44.240178 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b71227db-4083-4faf-9da1-50d06df620ea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-16466f-5pgqc\" (UID: \"b71227db-4083-4faf-9da1-50d06df620ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-16466f-5pgqc" Jul 23 00:51:44.341204 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:44.341165 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b71227db-4083-4faf-9da1-50d06df620ea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-16466f-5pgqc\" (UID: \"b71227db-4083-4faf-9da1-50d06df620ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-16466f-5pgqc" Jul 23 00:51:44.341923 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:44.341895 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b71227db-4083-4faf-9da1-50d06df620ea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-16466f-5pgqc\" (UID: \"b71227db-4083-4faf-9da1-50d06df620ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-16466f-5pgqc" Jul 23 00:51:44.447367 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:44.447261 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-16466f-5pgqc" Jul 23 00:51:44.630540 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:44.630513 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-16466f-5pgqc"] Jul 23 00:51:44.633504 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:51:44.633473 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb71227db_4083_4faf_9da1_50d06df620ea.slice/crio-7b7ff096598a2b586b93a2846ca48aed8df7147ba583bb5ffe8f378b8bdb0abd WatchSource:0}: Error finding container 7b7ff096598a2b586b93a2846ca48aed8df7147ba583bb5ffe8f378b8bdb0abd: Status 404 returned error can't find the container with id 7b7ff096598a2b586b93a2846ca48aed8df7147ba583bb5ffe8f378b8bdb0abd Jul 23 00:51:45.577594 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:45.577549 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-16466f-5pgqc" event={"ID":"b71227db-4083-4faf-9da1-50d06df620ea","Type":"ContainerStarted","Data":"685eefb0038fb11154159c14809ac68058c02bbfda8a0adfa168346ffcb9fd89"} Jul 23 00:51:45.578023 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:45.577654 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-16466f-5pgqc" event={"ID":"b71227db-4083-4faf-9da1-50d06df620ea","Type":"ContainerStarted","Data":"7b7ff096598a2b586b93a2846ca48aed8df7147ba583bb5ffe8f378b8bdb0abd"} Jul 23 00:51:45.594857 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:45.594800 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-16466f-5pgqc" podStartSLOduration=1.594749716 podStartE2EDuration="1.594749716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:51:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:51:45.594254802 +0000 UTC m=+2619.389761682" watchObservedRunningTime="2026-07-23 00:51:45.594749716 +0000 UTC m=+2619.390256586" Jul 23 00:51:46.493338 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:46.493293 2562 patch_prober.go:28] interesting pod/e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 23 00:51:46.493550 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:46.493358 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd" podUID="57c593fd-2b9c-4640-a49f-e70809c6e461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 23 00:51:46.585617 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:46.585580 2562 generic.go:356] "Generic (PLEG): container finished" podID="b71227db-4083-4faf-9da1-50d06df620ea" containerID="685eefb0038fb11154159c14809ac68058c02bbfda8a0adfa168346ffcb9fd89" exitCode=0 Jul 23 00:51:46.586055 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:46.585672 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-16466f-5pgqc" event={"ID":"b71227db-4083-4faf-9da1-50d06df620ea","Type":"ContainerDied","Data":"685eefb0038fb11154159c14809ac68058c02bbfda8a0adfa168346ffcb9fd89"} Jul 23 00:51:47.734613 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:47.734584 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-16466f-5pgqc" Jul 23 00:51:47.879155 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:47.879054 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b71227db-4083-4faf-9da1-50d06df620ea-maas-api-service-ca\") pod \"b71227db-4083-4faf-9da1-50d06df620ea\" (UID: \"b71227db-4083-4faf-9da1-50d06df620ea\") " Jul 23 00:51:47.879471 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:47.879444 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b71227db-4083-4faf-9da1-50d06df620ea-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b71227db-4083-4faf-9da1-50d06df620ea" (UID: "b71227db-4083-4faf-9da1-50d06df620ea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:51:47.980300 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:47.980257 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b71227db-4083-4faf-9da1-50d06df620ea-maas-api-service-ca\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:51:48.599958 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:48.599866 2562 generic.go:356] "Generic (PLEG): container finished" podID="57c593fd-2b9c-4640-a49f-e70809c6e461" containerID="0724f19a60dd45ab1c78e47b7da3abd426feaab05ec477cf3f2f40be0c121722" exitCode=0 Jul 23 00:51:48.600162 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:48.599953 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd" event={"ID":"57c593fd-2b9c-4640-a49f-e70809c6e461","Type":"ContainerDied","Data":"0724f19a60dd45ab1c78e47b7da3abd426feaab05ec477cf3f2f40be0c121722"} Jul 23 00:51:48.600455 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:48.600431 2562 scope.go:117] "RemoveContainer" containerID="0724f19a60dd45ab1c78e47b7da3abd426feaab05ec477cf3f2f40be0c121722" Jul 23 00:51:48.601563 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:48.601531 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-16466f-5pgqc" event={"ID":"b71227db-4083-4faf-9da1-50d06df620ea","Type":"ContainerDied","Data":"7b7ff096598a2b586b93a2846ca48aed8df7147ba583bb5ffe8f378b8bdb0abd"} Jul 23 00:51:48.601656 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:48.601568 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:51:48.601656 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:48.601635 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:51:48.601656 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:48.601583 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b7ff096598a2b586b93a2846ca48aed8df7147ba583bb5ffe8f378b8bdb0abd" Jul 23 00:51:48.601806 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:48.601668 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:51:48.601806 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:48.601549 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-16466f-5pgqc" Jul 23 00:51:49.609062 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:49.609024 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd" event={"ID":"57c593fd-2b9c-4640-a49f-e70809c6e461","Type":"ContainerStarted","Data":"2025ef62f8a3ff5a1e91764165d697855c164f9650b549a4fa49d7d907ba494f"} Jul 23 00:51:50.044117 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:50.044074 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd" Jul 23 00:51:50.044348 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:50.044321 2562 patch_prober.go:28] interesting pod/e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 23 00:51:50.044402 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:50.044382 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd" podUID="57c593fd-2b9c-4640-a49f-e70809c6e461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 23 00:51:51.043729 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:51.043693 2562 patch_prober.go:28] interesting pod/e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 23 00:51:51.044140 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:51.043775 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd" podUID="57c593fd-2b9c-4640-a49f-e70809c6e461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 23 00:51:52.044169 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:52.044127 2562 patch_prober.go:28] interesting pod/e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 23 00:51:52.044664 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:52.044195 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd" podUID="57c593fd-2b9c-4640-a49f-e70809c6e461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 23 00:51:53.043302 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:53.043268 2562 patch_prober.go:28] interesting pod/e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 23 00:51:53.043483 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:53.043327 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd" podUID="57c593fd-2b9c-4640-a49f-e70809c6e461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 23 00:51:54.043981 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:54.043935 2562 patch_prober.go:28] interesting pod/e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 23 00:51:54.044411 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:54.044007 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd" podUID="57c593fd-2b9c-4640-a49f-e70809c6e461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 23 00:51:55.047695 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:55.047657 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd" Jul 23 00:51:55.048177 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:55.047936 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd" Jul 23 00:51:55.048804 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:51:55.048784 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd" Jul 23 00:52:04.222310 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:04.222265 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-16466f-5pgqc"] Jul 23 00:52:04.231133 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:04.231099 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-16466f-5pgqc"] Jul 23 00:52:05.006750 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:05.006704 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b71227db-4083-4faf-9da1-50d06df620ea" path="/var/lib/kubelet/pods/b71227db-4083-4faf-9da1-50d06df620ea/volumes" Jul 23 00:52:11.323640 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:11.323596 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e2debd-2bjdz"] Jul 23 00:52:11.324198 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:11.324019 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b71227db-4083-4faf-9da1-50d06df620ea" containerName="revoke-keys" Jul 23 00:52:11.324198 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:11.324032 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b71227db-4083-4faf-9da1-50d06df620ea" containerName="revoke-keys" Jul 23 00:52:11.324198 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:11.324098 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b71227db-4083-4faf-9da1-50d06df620ea" containerName="revoke-keys" Jul 23 00:52:11.327237 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:11.327213 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e2debd-2bjdz" Jul 23 00:52:11.330432 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:11.330410 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w8tk2\"" Jul 23 00:52:11.335943 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:11.335709 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e2debd-2bjdz"] Jul 23 00:52:11.420950 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:11.420908 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1820433-a30d-44f6-bbb5-53128e1e3d24-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e2debd-2bjdz\" (UID: \"f1820433-a30d-44f6-bbb5-53128e1e3d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e2debd-2bjdz" Jul 23 00:52:11.522170 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:11.522130 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1820433-a30d-44f6-bbb5-53128e1e3d24-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e2debd-2bjdz\" (UID: \"f1820433-a30d-44f6-bbb5-53128e1e3d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e2debd-2bjdz" Jul 23 00:52:11.522881 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:11.522859 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1820433-a30d-44f6-bbb5-53128e1e3d24-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e2debd-2bjdz\" (UID: \"f1820433-a30d-44f6-bbb5-53128e1e3d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e2debd-2bjdz" Jul 23 00:52:11.640677 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:11.640556 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e2debd-2bjdz" Jul 23 00:52:11.797435 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:11.797405 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e2debd-2bjdz"] Jul 23 00:52:11.799318 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:52:11.799288 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1820433_a30d_44f6_bbb5_53128e1e3d24.slice/crio-a27e6fdb0fcae33a5282eb8d6f144f2ea5f98c606e4c167e7b5ac3cd6ac6fc5d WatchSource:0}: Error finding container a27e6fdb0fcae33a5282eb8d6f144f2ea5f98c606e4c167e7b5ac3cd6ac6fc5d: Status 404 returned error can't find the container with id a27e6fdb0fcae33a5282eb8d6f144f2ea5f98c606e4c167e7b5ac3cd6ac6fc5d Jul 23 00:52:12.750018 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:12.749987 2562 generic.go:356] "Generic (PLEG): container finished" podID="f1820433-a30d-44f6-bbb5-53128e1e3d24" containerID="8113b33143dedcbc72e7e48ebfdac464ff1eebe55d8e05648b9644b4d365da09" exitCode=0 Jul 23 00:52:12.750442 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:12.750064 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e2debd-2bjdz" event={"ID":"f1820433-a30d-44f6-bbb5-53128e1e3d24","Type":"ContainerDied","Data":"8113b33143dedcbc72e7e48ebfdac464ff1eebe55d8e05648b9644b4d365da09"} Jul 23 00:52:12.750442 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:12.750108 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e2debd-2bjdz" event={"ID":"f1820433-a30d-44f6-bbb5-53128e1e3d24","Type":"ContainerStarted","Data":"a27e6fdb0fcae33a5282eb8d6f144f2ea5f98c606e4c167e7b5ac3cd6ac6fc5d"} Jul 23 00:52:13.894370 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:13.894342 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e2debd-2bjdz" Jul 23 00:52:13.946473 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:13.946436 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1820433-a30d-44f6-bbb5-53128e1e3d24-maas-api-service-ca\") pod \"f1820433-a30d-44f6-bbb5-53128e1e3d24\" (UID: \"f1820433-a30d-44f6-bbb5-53128e1e3d24\") " Jul 23 00:52:13.946898 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:13.946872 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1820433-a30d-44f6-bbb5-53128e1e3d24-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f1820433-a30d-44f6-bbb5-53128e1e3d24" (UID: "f1820433-a30d-44f6-bbb5-53128e1e3d24"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:52:14.047351 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:14.047304 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1820433-a30d-44f6-bbb5-53128e1e3d24-maas-api-service-ca\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:52:14.770022 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:14.769984 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e2debd-2bjdz" Jul 23 00:52:14.770022 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:14.770015 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e2debd-2bjdz" event={"ID":"f1820433-a30d-44f6-bbb5-53128e1e3d24","Type":"ContainerDied","Data":"a27e6fdb0fcae33a5282eb8d6f144f2ea5f98c606e4c167e7b5ac3cd6ac6fc5d"} Jul 23 00:52:14.770247 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:14.770089 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a27e6fdb0fcae33a5282eb8d6f144f2ea5f98c606e4c167e7b5ac3cd6ac6fc5d" Jul 23 00:52:31.415394 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:31.415350 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e2debd-2bjdz"] Jul 23 00:52:31.420842 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:31.420804 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e2debd-2bjdz"] Jul 23 00:52:33.007979 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:33.007939 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1820433-a30d-44f6-bbb5-53128e1e3d24" path="/var/lib/kubelet/pods/f1820433-a30d-44f6-bbb5-53128e1e3d24/volumes" Jul 23 00:52:37.881980 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:37.881934 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd"] Jul 23 00:52:37.882591 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:37.882152 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd" podUID="57c593fd-2b9c-4640-a49f-e70809c6e461" containerName="istio-proxy" containerID="cri-o://2025ef62f8a3ff5a1e91764165d697855c164f9650b549a4fa49d7d907ba494f" gracePeriod=30 Jul 23 00:52:40.050969 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:40.050923 2562 patch_prober.go:28] interesting pod/e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 23 00:52:40.051388 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:40.050986 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd" podUID="57c593fd-2b9c-4640-a49f-e70809c6e461" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 23 00:52:43.153740 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:43.153668 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd" Jul 23 00:52:43.220389 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:43.220345 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57c593fd-2b9c-4640-a49f-e70809c6e461-istio-envoy\") pod \"57c593fd-2b9c-4640-a49f-e70809c6e461\" (UID: \"57c593fd-2b9c-4640-a49f-e70809c6e461\") " Jul 23 00:52:43.220572 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:43.220408 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57c593fd-2b9c-4640-a49f-e70809c6e461-istiod-ca-cert\") pod \"57c593fd-2b9c-4640-a49f-e70809c6e461\" (UID: \"57c593fd-2b9c-4640-a49f-e70809c6e461\") " Jul 23 00:52:43.220572 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:43.220452 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57c593fd-2b9c-4640-a49f-e70809c6e461-istio-data\") pod \"57c593fd-2b9c-4640-a49f-e70809c6e461\" (UID: \"57c593fd-2b9c-4640-a49f-e70809c6e461\") " Jul 23 00:52:43.220572 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:43.220470 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4dggb\" (UniqueName: \"kubernetes.io/projected/57c593fd-2b9c-4640-a49f-e70809c6e461-kube-api-access-4dggb\") pod \"57c593fd-2b9c-4640-a49f-e70809c6e461\" (UID: \"57c593fd-2b9c-4640-a49f-e70809c6e461\") " Jul 23 00:52:43.220572 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:43.220489 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57c593fd-2b9c-4640-a49f-e70809c6e461-workload-socket\") pod \"57c593fd-2b9c-4640-a49f-e70809c6e461\" (UID: \"57c593fd-2b9c-4640-a49f-e70809c6e461\") " Jul 23 00:52:43.220572 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:43.220549 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57c593fd-2b9c-4640-a49f-e70809c6e461-credential-socket\") pod \"57c593fd-2b9c-4640-a49f-e70809c6e461\" (UID: \"57c593fd-2b9c-4640-a49f-e70809c6e461\") " Jul 23 00:52:43.220872 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:43.220573 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57c593fd-2b9c-4640-a49f-e70809c6e461-istio-podinfo\") pod \"57c593fd-2b9c-4640-a49f-e70809c6e461\" (UID: \"57c593fd-2b9c-4640-a49f-e70809c6e461\") " Jul 23 00:52:43.220872 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:43.220620 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57c593fd-2b9c-4640-a49f-e70809c6e461-workload-certs\") pod \"57c593fd-2b9c-4640-a49f-e70809c6e461\" (UID: \"57c593fd-2b9c-4640-a49f-e70809c6e461\") " Jul 23 00:52:43.220872 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:43.220651 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57c593fd-2b9c-4640-a49f-e70809c6e461-istio-token\") pod \"57c593fd-2b9c-4640-a49f-e70809c6e461\" (UID: \"57c593fd-2b9c-4640-a49f-e70809c6e461\") " Jul 23 00:52:43.220872 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:43.220853 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57c593fd-2b9c-4640-a49f-e70809c6e461-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "57c593fd-2b9c-4640-a49f-e70809c6e461" (UID: "57c593fd-2b9c-4640-a49f-e70809c6e461"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:52:43.221096 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:43.220861 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57c593fd-2b9c-4640-a49f-e70809c6e461-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "57c593fd-2b9c-4640-a49f-e70809c6e461" (UID: "57c593fd-2b9c-4640-a49f-e70809c6e461"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:52:43.221096 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:43.221018 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57c593fd-2b9c-4640-a49f-e70809c6e461-istiod-ca-cert\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:52:43.221096 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:43.221039 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57c593fd-2b9c-4640-a49f-e70809c6e461-credential-socket\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:52:43.221096 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:43.221039 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57c593fd-2b9c-4640-a49f-e70809c6e461-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "57c593fd-2b9c-4640-a49f-e70809c6e461" (UID: "57c593fd-2b9c-4640-a49f-e70809c6e461"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:52:43.221096 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:43.221085 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57c593fd-2b9c-4640-a49f-e70809c6e461-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "57c593fd-2b9c-4640-a49f-e70809c6e461" (UID: "57c593fd-2b9c-4640-a49f-e70809c6e461"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:52:43.221594 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:43.221565 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57c593fd-2b9c-4640-a49f-e70809c6e461-istio-data" (OuterVolumeSpecName: "istio-data") pod "57c593fd-2b9c-4640-a49f-e70809c6e461" (UID: "57c593fd-2b9c-4640-a49f-e70809c6e461"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:52:43.222855 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:43.222823 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57c593fd-2b9c-4640-a49f-e70809c6e461-kube-api-access-4dggb" (OuterVolumeSpecName: "kube-api-access-4dggb") pod "57c593fd-2b9c-4640-a49f-e70809c6e461" (UID: "57c593fd-2b9c-4640-a49f-e70809c6e461"). InnerVolumeSpecName "kube-api-access-4dggb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:52:43.222961 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:43.222916 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57c593fd-2b9c-4640-a49f-e70809c6e461-istio-token" (OuterVolumeSpecName: "istio-token") pod "57c593fd-2b9c-4640-a49f-e70809c6e461" (UID: "57c593fd-2b9c-4640-a49f-e70809c6e461"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:52:43.223084 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:43.223057 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57c593fd-2b9c-4640-a49f-e70809c6e461-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "57c593fd-2b9c-4640-a49f-e70809c6e461" (UID: "57c593fd-2b9c-4640-a49f-e70809c6e461"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:52:43.223122 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:43.223106 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57c593fd-2b9c-4640-a49f-e70809c6e461-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "57c593fd-2b9c-4640-a49f-e70809c6e461" (UID: "57c593fd-2b9c-4640-a49f-e70809c6e461"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 00:52:43.322444 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:43.322401 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57c593fd-2b9c-4640-a49f-e70809c6e461-istio-data\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:52:43.322444 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:43.322435 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dggb\" (UniqueName: \"kubernetes.io/projected/57c593fd-2b9c-4640-a49f-e70809c6e461-kube-api-access-4dggb\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:52:43.322444 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:43.322446 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57c593fd-2b9c-4640-a49f-e70809c6e461-workload-socket\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:52:43.322444 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:43.322455 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57c593fd-2b9c-4640-a49f-e70809c6e461-istio-podinfo\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:52:43.322724 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:43.322464 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57c593fd-2b9c-4640-a49f-e70809c6e461-workload-certs\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:52:43.322724 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:43.322471 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57c593fd-2b9c-4640-a49f-e70809c6e461-istio-token\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:52:43.322724 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:43.322479 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57c593fd-2b9c-4640-a49f-e70809c6e461-istio-envoy\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:52:43.953365 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:43.953328 2562 generic.go:356] "Generic (PLEG): container finished" podID="57c593fd-2b9c-4640-a49f-e70809c6e461" containerID="2025ef62f8a3ff5a1e91764165d697855c164f9650b549a4fa49d7d907ba494f" exitCode=0 Jul 23 00:52:43.953520 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:43.953380 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd" Jul 23 00:52:43.953520 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:43.953411 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd" event={"ID":"57c593fd-2b9c-4640-a49f-e70809c6e461","Type":"ContainerDied","Data":"2025ef62f8a3ff5a1e91764165d697855c164f9650b549a4fa49d7d907ba494f"} Jul 23 00:52:43.953520 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:43.953468 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd" event={"ID":"57c593fd-2b9c-4640-a49f-e70809c6e461","Type":"ContainerDied","Data":"ee96c98f0898df316df56ff0e1f973be6efb1c76ec3b39dddffeaeb2eabec5b1"} Jul 23 00:52:43.953520 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:43.953491 2562 scope.go:117] "RemoveContainer" containerID="2025ef62f8a3ff5a1e91764165d697855c164f9650b549a4fa49d7d907ba494f" Jul 23 00:52:43.963706 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:43.963679 2562 scope.go:117] "RemoveContainer" containerID="0724f19a60dd45ab1c78e47b7da3abd426feaab05ec477cf3f2f40be0c121722" Jul 23 00:52:43.972573 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:43.972549 2562 scope.go:117] "RemoveContainer" containerID="2025ef62f8a3ff5a1e91764165d697855c164f9650b549a4fa49d7d907ba494f" Jul 23 00:52:43.972949 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:52:43.972927 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2025ef62f8a3ff5a1e91764165d697855c164f9650b549a4fa49d7d907ba494f\": container with ID starting with 2025ef62f8a3ff5a1e91764165d697855c164f9650b549a4fa49d7d907ba494f not found: ID does not exist" containerID="2025ef62f8a3ff5a1e91764165d697855c164f9650b549a4fa49d7d907ba494f" Jul 23 00:52:43.973033 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:43.972960 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2025ef62f8a3ff5a1e91764165d697855c164f9650b549a4fa49d7d907ba494f"} err="failed to get container status \"2025ef62f8a3ff5a1e91764165d697855c164f9650b549a4fa49d7d907ba494f\": rpc error: code = NotFound desc = could not find container \"2025ef62f8a3ff5a1e91764165d697855c164f9650b549a4fa49d7d907ba494f\": container with ID starting with 2025ef62f8a3ff5a1e91764165d697855c164f9650b549a4fa49d7d907ba494f not found: ID does not exist" Jul 23 00:52:43.973033 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:43.972980 2562 scope.go:117] "RemoveContainer" containerID="0724f19a60dd45ab1c78e47b7da3abd426feaab05ec477cf3f2f40be0c121722" Jul 23 00:52:43.973248 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:52:43.973228 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0724f19a60dd45ab1c78e47b7da3abd426feaab05ec477cf3f2f40be0c121722\": container with ID starting with 0724f19a60dd45ab1c78e47b7da3abd426feaab05ec477cf3f2f40be0c121722 not found: ID does not exist" containerID="0724f19a60dd45ab1c78e47b7da3abd426feaab05ec477cf3f2f40be0c121722" Jul 23 00:52:43.973294 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:43.973254 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0724f19a60dd45ab1c78e47b7da3abd426feaab05ec477cf3f2f40be0c121722"} err="failed to get container status \"0724f19a60dd45ab1c78e47b7da3abd426feaab05ec477cf3f2f40be0c121722\": rpc error: code = NotFound desc = could not find container \"0724f19a60dd45ab1c78e47b7da3abd426feaab05ec477cf3f2f40be0c121722\": container with ID starting with 0724f19a60dd45ab1c78e47b7da3abd426feaab05ec477cf3f2f40be0c121722 not found: ID does not exist" Jul 23 00:52:43.978180 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:43.978147 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd"] Jul 23 00:52:43.982946 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:43.982913 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-e2debd-openshift-default-5df6d6dcdf-622wd"] Jul 23 00:52:45.007675 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:45.007640 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57c593fd-2b9c-4640-a49f-e70809c6e461" path="/var/lib/kubelet/pods/57c593fd-2b9c-4640-a49f-e70809c6e461/volumes" Jul 23 00:52:46.933543 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:46.933508 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz"] Jul 23 00:52:46.934030 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:46.934011 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57c593fd-2b9c-4640-a49f-e70809c6e461" containerName="istio-proxy" Jul 23 00:52:46.934030 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:46.934030 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="57c593fd-2b9c-4640-a49f-e70809c6e461" containerName="istio-proxy" Jul 23 00:52:46.934177 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:46.934039 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1820433-a30d-44f6-bbb5-53128e1e3d24" containerName="revoke-keys" Jul 23 00:52:46.934177 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:46.934046 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1820433-a30d-44f6-bbb5-53128e1e3d24" containerName="revoke-keys" Jul 23 00:52:46.934177 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:46.934154 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="57c593fd-2b9c-4640-a49f-e70809c6e461" containerName="istio-proxy" Jul 23 00:52:46.934177 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:46.934171 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1820433-a30d-44f6-bbb5-53128e1e3d24" containerName="revoke-keys" Jul 23 00:52:46.934318 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:46.934183 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="57c593fd-2b9c-4640-a49f-e70809c6e461" containerName="istio-proxy" Jul 23 00:52:46.938790 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:46.938747 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz" Jul 23 00:52:46.943533 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:46.943507 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-d29ed6-openshift-default-dockercfg-gp5mq\"" Jul 23 00:52:46.956259 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:46.956225 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz"] Jul 23 00:52:47.055999 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:47.055961 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4afd8bba-2465-4d15-a7b8-4550cfd379f3-workload-certs\") pod \"e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz\" (UID: \"4afd8bba-2465-4d15-a7b8-4550cfd379f3\") " pod="openshift-ingress/e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz" Jul 23 00:52:47.055999 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:47.055998 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4afd8bba-2465-4d15-a7b8-4550cfd379f3-istio-data\") pod \"e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz\" (UID: \"4afd8bba-2465-4d15-a7b8-4550cfd379f3\") " pod="openshift-ingress/e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz" Jul 23 00:52:47.055999 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:47.056023 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4afd8bba-2465-4d15-a7b8-4550cfd379f3-istiod-ca-cert\") pod \"e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz\" (UID: \"4afd8bba-2465-4d15-a7b8-4550cfd379f3\") " pod="openshift-ingress/e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz" Jul 23 00:52:47.056325 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:47.056145 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4afd8bba-2465-4d15-a7b8-4550cfd379f3-credential-socket\") pod \"e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz\" (UID: \"4afd8bba-2465-4d15-a7b8-4550cfd379f3\") " pod="openshift-ingress/e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz" Jul 23 00:52:47.056325 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:47.056191 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4afd8bba-2465-4d15-a7b8-4550cfd379f3-istio-token\") pod \"e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz\" (UID: \"4afd8bba-2465-4d15-a7b8-4550cfd379f3\") " pod="openshift-ingress/e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz" Jul 23 00:52:47.056325 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:47.056227 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4afd8bba-2465-4d15-a7b8-4550cfd379f3-istio-podinfo\") pod \"e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz\" (UID: \"4afd8bba-2465-4d15-a7b8-4550cfd379f3\") " pod="openshift-ingress/e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz" Jul 23 00:52:47.056325 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:47.056248 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8sw5\" (UniqueName: \"kubernetes.io/projected/4afd8bba-2465-4d15-a7b8-4550cfd379f3-kube-api-access-d8sw5\") pod \"e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz\" (UID: \"4afd8bba-2465-4d15-a7b8-4550cfd379f3\") " pod="openshift-ingress/e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz" Jul 23 00:52:47.056325 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:47.056288 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4afd8bba-2465-4d15-a7b8-4550cfd379f3-istio-envoy\") pod \"e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz\" (UID: \"4afd8bba-2465-4d15-a7b8-4550cfd379f3\") " pod="openshift-ingress/e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz" Jul 23 00:52:47.056325 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:47.056323 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4afd8bba-2465-4d15-a7b8-4550cfd379f3-workload-socket\") pod \"e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz\" (UID: \"4afd8bba-2465-4d15-a7b8-4550cfd379f3\") " pod="openshift-ingress/e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz" Jul 23 00:52:47.157669 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:47.157636 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4afd8bba-2465-4d15-a7b8-4550cfd379f3-workload-certs\") pod \"e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz\" (UID: \"4afd8bba-2465-4d15-a7b8-4550cfd379f3\") " pod="openshift-ingress/e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz" Jul 23 00:52:47.157669 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:47.157672 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4afd8bba-2465-4d15-a7b8-4550cfd379f3-istio-data\") pod \"e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz\" (UID: \"4afd8bba-2465-4d15-a7b8-4550cfd379f3\") " pod="openshift-ingress/e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz" Jul 23 00:52:47.157980 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:47.157695 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4afd8bba-2465-4d15-a7b8-4550cfd379f3-istiod-ca-cert\") pod \"e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz\" (UID: \"4afd8bba-2465-4d15-a7b8-4550cfd379f3\") " pod="openshift-ingress/e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz" Jul 23 00:52:47.157980 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:47.157775 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4afd8bba-2465-4d15-a7b8-4550cfd379f3-credential-socket\") pod \"e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz\" (UID: \"4afd8bba-2465-4d15-a7b8-4550cfd379f3\") " pod="openshift-ingress/e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz" Jul 23 00:52:47.157980 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:47.157808 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4afd8bba-2465-4d15-a7b8-4550cfd379f3-istio-token\") pod \"e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz\" (UID: \"4afd8bba-2465-4d15-a7b8-4550cfd379f3\") " pod="openshift-ingress/e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz" Jul 23 00:52:47.157980 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:47.157893 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4afd8bba-2465-4d15-a7b8-4550cfd379f3-istio-podinfo\") pod \"e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz\" (UID: \"4afd8bba-2465-4d15-a7b8-4550cfd379f3\") " pod="openshift-ingress/e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz" Jul 23 00:52:47.157980 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:47.157947 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8sw5\" (UniqueName: \"kubernetes.io/projected/4afd8bba-2465-4d15-a7b8-4550cfd379f3-kube-api-access-d8sw5\") pod \"e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz\" (UID: \"4afd8bba-2465-4d15-a7b8-4550cfd379f3\") " pod="openshift-ingress/e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz" Jul 23 00:52:47.158251 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:47.157994 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4afd8bba-2465-4d15-a7b8-4550cfd379f3-istio-envoy\") pod \"e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz\" (UID: \"4afd8bba-2465-4d15-a7b8-4550cfd379f3\") " pod="openshift-ingress/e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz" Jul 23 00:52:47.158251 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:47.158033 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4afd8bba-2465-4d15-a7b8-4550cfd379f3-workload-socket\") pod \"e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz\" (UID: \"4afd8bba-2465-4d15-a7b8-4550cfd379f3\") " pod="openshift-ingress/e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz" Jul 23 00:52:47.158251 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:47.158137 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4afd8bba-2465-4d15-a7b8-4550cfd379f3-workload-certs\") pod \"e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz\" (UID: \"4afd8bba-2465-4d15-a7b8-4550cfd379f3\") " pod="openshift-ingress/e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz" Jul 23 00:52:47.158251 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:47.158171 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4afd8bba-2465-4d15-a7b8-4550cfd379f3-istio-data\") pod \"e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz\" (UID: \"4afd8bba-2465-4d15-a7b8-4550cfd379f3\") " pod="openshift-ingress/e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz" Jul 23 00:52:47.158251 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:47.158231 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4afd8bba-2465-4d15-a7b8-4550cfd379f3-credential-socket\") pod \"e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz\" (UID: \"4afd8bba-2465-4d15-a7b8-4550cfd379f3\") " pod="openshift-ingress/e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz" Jul 23 00:52:47.158579 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:47.158410 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4afd8bba-2465-4d15-a7b8-4550cfd379f3-workload-socket\") pod \"e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz\" (UID: \"4afd8bba-2465-4d15-a7b8-4550cfd379f3\") " pod="openshift-ingress/e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz" Jul 23 00:52:47.158852 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:47.158829 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4afd8bba-2465-4d15-a7b8-4550cfd379f3-istiod-ca-cert\") pod \"e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz\" (UID: \"4afd8bba-2465-4d15-a7b8-4550cfd379f3\") " pod="openshift-ingress/e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz" Jul 23 00:52:47.160243 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:47.160221 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4afd8bba-2465-4d15-a7b8-4550cfd379f3-istio-envoy\") pod \"e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz\" (UID: \"4afd8bba-2465-4d15-a7b8-4550cfd379f3\") " pod="openshift-ingress/e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz" Jul 23 00:52:47.160457 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:47.160438 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4afd8bba-2465-4d15-a7b8-4550cfd379f3-istio-podinfo\") pod \"e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz\" (UID: \"4afd8bba-2465-4d15-a7b8-4550cfd379f3\") " pod="openshift-ingress/e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz" Jul 23 00:52:47.166039 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:47.166015 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4afd8bba-2465-4d15-a7b8-4550cfd379f3-istio-token\") pod \"e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz\" (UID: \"4afd8bba-2465-4d15-a7b8-4550cfd379f3\") " pod="openshift-ingress/e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz" Jul 23 00:52:47.166167 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:47.166154 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8sw5\" (UniqueName: \"kubernetes.io/projected/4afd8bba-2465-4d15-a7b8-4550cfd379f3-kube-api-access-d8sw5\") pod \"e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz\" (UID: \"4afd8bba-2465-4d15-a7b8-4550cfd379f3\") " pod="openshift-ingress/e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz" Jul 23 00:52:47.249972 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:47.249876 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz" Jul 23 00:52:47.402806 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:47.402734 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz"] Jul 23 00:52:47.406074 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:52:47.406042 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4afd8bba_2465_4d15_a7b8_4550cfd379f3.slice/crio-1fef00766007da37b43b197591a2dab05816ebf6c775e8ef86a6f69eb660e95a WatchSource:0}: Error finding container 1fef00766007da37b43b197591a2dab05816ebf6c775e8ef86a6f69eb660e95a: Status 404 returned error can't find the container with id 1fef00766007da37b43b197591a2dab05816ebf6c775e8ef86a6f69eb660e95a Jul 23 00:52:47.408437 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:47.408395 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:52:47.408575 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:47.408476 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:52:47.408575 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:47.408510 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:52:47.986276 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:47.986227 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz" event={"ID":"4afd8bba-2465-4d15-a7b8-4550cfd379f3","Type":"ContainerStarted","Data":"9618eced548337b615a501148a663baf4232bf90f11aa2d3b64e6d4cbf4da837"} Jul 23 00:52:47.986807 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:47.986316 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz" event={"ID":"4afd8bba-2465-4d15-a7b8-4550cfd379f3","Type":"ContainerStarted","Data":"1fef00766007da37b43b197591a2dab05816ebf6c775e8ef86a6f69eb660e95a"} Jul 23 00:52:48.013257 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:48.013194 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz" podStartSLOduration=2.013177327 podStartE2EDuration="2.013177327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:52:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:52:48.010409416 +0000 UTC m=+2681.805916284" watchObservedRunningTime="2026-07-23 00:52:48.013177327 +0000 UTC m=+2681.808684183" Jul 23 00:52:48.250741 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:48.250637 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz" Jul 23 00:52:48.255850 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:48.255816 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz" Jul 23 00:52:48.992009 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:48.991980 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz" Jul 23 00:52:48.993096 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:52:48.993072 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz" Jul 23 00:53:02.069829 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:53:02.069791 2562 generic.go:356] "Generic (PLEG): container finished" podID="4afd8bba-2465-4d15-a7b8-4550cfd379f3" containerID="9618eced548337b615a501148a663baf4232bf90f11aa2d3b64e6d4cbf4da837" exitCode=0 Jul 23 00:53:02.070327 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:53:02.069844 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz" event={"ID":"4afd8bba-2465-4d15-a7b8-4550cfd379f3","Type":"ContainerDied","Data":"9618eced548337b615a501148a663baf4232bf90f11aa2d3b64e6d4cbf4da837"} Jul 23 00:53:02.070399 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:53:02.070331 2562 scope.go:117] "RemoveContainer" containerID="9618eced548337b615a501148a663baf4232bf90f11aa2d3b64e6d4cbf4da837" Jul 23 00:53:02.071345 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:53:02.071320 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:53:02.071410 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:53:02.071391 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:53:02.071467 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:53:02.071454 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 00:53:03.077124 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:53:03.077086 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz" event={"ID":"4afd8bba-2465-4d15-a7b8-4550cfd379f3","Type":"ContainerStarted","Data":"3eab1e1cb3ff3ed8ea83b6638e6b2294ed99538bb543c64aef6d443e71005bf0"} Jul 23 00:53:03.250030 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:53:03.249988 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz" Jul 23 00:53:03.251354 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:53:03.251325 2562 patch_prober.go:28] interesting pod/e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 23 00:53:03.251475 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:53:03.251383 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz" podUID="4afd8bba-2465-4d15-a7b8-4550cfd379f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 23 00:53:03.992260 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:53:03.992222 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz" Jul 23 00:53:04.250477 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:53:04.250389 2562 patch_prober.go:28] interesting pod/e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 23 00:53:04.250477 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:53:04.250462 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz" podUID="4afd8bba-2465-4d15-a7b8-4550cfd379f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 23 00:53:05.250398 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:53:05.250360 2562 patch_prober.go:28] interesting pod/e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 23 00:53:05.250583 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:53:05.250415 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz" podUID="4afd8bba-2465-4d15-a7b8-4550cfd379f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 23 00:53:06.250471 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:53:06.250430 2562 patch_prober.go:28] interesting pod/e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 23 00:53:06.250660 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:53:06.250485 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz" podUID="4afd8bba-2465-4d15-a7b8-4550cfd379f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 23 00:53:07.250695 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:53:07.250649 2562 patch_prober.go:28] interesting pod/e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 23 00:53:07.251162 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:53:07.250718 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz" podUID="4afd8bba-2465-4d15-a7b8-4550cfd379f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 23 00:53:08.254601 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:53:08.254562 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz" Jul 23 00:53:08.255538 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:53:08.255516 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz" Jul 23 00:58:07.204927 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:07.204893 2562 scope.go:117] "RemoveContainer" containerID="685eefb0038fb11154159c14809ac68058c02bbfda8a0adfa168346ffcb9fd89" Jul 23 00:58:40.053555 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:40.053516 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7852-140045906377648"] Jul 23 00:58:40.054063 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:40.053945 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57c593fd-2b9c-4640-a49f-e70809c6e461" containerName="istio-proxy" Jul 23 00:58:40.054063 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:40.053960 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="57c593fd-2b9c-4640-a49f-e70809c6e461" containerName="istio-proxy" Jul 23 00:58:40.057150 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:40.057130 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7852-140045906377648" Jul 23 00:58:40.060329 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:40.060302 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-wjlfl\"" Jul 23 00:58:40.064618 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:40.064585 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7852-140045906377648"] Jul 23 00:58:40.066168 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:40.066136 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlvpl\" (UniqueName: \"kubernetes.io/projected/baf50b51-6675-4e84-8793-40129e8abf91-kube-api-access-vlvpl\") pod \"test-curl-7852-140045906377648\" (UID: \"baf50b51-6675-4e84-8793-40129e8abf91\") " pod="opendatahub/test-curl-7852-140045906377648" Jul 23 00:58:40.167249 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:40.167209 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlvpl\" (UniqueName: \"kubernetes.io/projected/baf50b51-6675-4e84-8793-40129e8abf91-kube-api-access-vlvpl\") pod \"test-curl-7852-140045906377648\" (UID: \"baf50b51-6675-4e84-8793-40129e8abf91\") " pod="opendatahub/test-curl-7852-140045906377648" Jul 23 00:58:40.176397 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:40.176363 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlvpl\" (UniqueName: \"kubernetes.io/projected/baf50b51-6675-4e84-8793-40129e8abf91-kube-api-access-vlvpl\") pod \"test-curl-7852-140045906377648\" (UID: \"baf50b51-6675-4e84-8793-40129e8abf91\") " pod="opendatahub/test-curl-7852-140045906377648" Jul 23 00:58:40.368142 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:40.368034 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7852-140045906377648" Jul 23 00:58:40.505388 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:40.505361 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7852-140045906377648"] Jul 23 00:58:40.507288 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:58:40.507251 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbaf50b51_6675_4e84_8793_40129e8abf91.slice/crio-a780960eb46f1405eee7d81cb8d0de4655a70254c58a836ad8a48ffeb305c4d6 WatchSource:0}: Error finding container a780960eb46f1405eee7d81cb8d0de4655a70254c58a836ad8a48ffeb305c4d6: Status 404 returned error can't find the container with id a780960eb46f1405eee7d81cb8d0de4655a70254c58a836ad8a48ffeb305c4d6 Jul 23 00:58:40.508910 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:40.508895 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 00:58:41.054797 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:41.054731 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7852-140045906377648" event={"ID":"baf50b51-6675-4e84-8793-40129e8abf91","Type":"ContainerStarted","Data":"a780960eb46f1405eee7d81cb8d0de4655a70254c58a836ad8a48ffeb305c4d6"} Jul 23 00:58:42.064281 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:42.064242 2562 generic.go:356] "Generic (PLEG): container finished" podID="baf50b51-6675-4e84-8793-40129e8abf91" containerID="cd1bbf596a61f1c7042016f4e0fce3bba9240e825bd5a9efcd431d4ec8258822" exitCode=0 Jul 23 00:58:42.064800 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:42.064336 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7852-140045906377648" event={"ID":"baf50b51-6675-4e84-8793-40129e8abf91","Type":"ContainerDied","Data":"cd1bbf596a61f1c7042016f4e0fce3bba9240e825bd5a9efcd431d4ec8258822"} Jul 23 00:58:43.211751 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:43.211723 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7852-140045906377648" Jul 23 00:58:43.251681 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:43.251625 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7852-140045906377648_baf50b51-6675-4e84-8793-40129e8abf91/test-curl-7852-140045906377648/0.log" Jul 23 00:58:43.279254 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:43.279209 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7852-140045906377648"] Jul 23 00:58:43.287947 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:43.287916 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7852-140045906377648"] Jul 23 00:58:43.296570 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:43.296546 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vlvpl\" (UniqueName: \"kubernetes.io/projected/baf50b51-6675-4e84-8793-40129e8abf91-kube-api-access-vlvpl\") pod \"baf50b51-6675-4e84-8793-40129e8abf91\" (UID: \"baf50b51-6675-4e84-8793-40129e8abf91\") " Jul 23 00:58:43.298815 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:43.298785 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baf50b51-6675-4e84-8793-40129e8abf91-kube-api-access-vlvpl" (OuterVolumeSpecName: "kube-api-access-vlvpl") pod "baf50b51-6675-4e84-8793-40129e8abf91" (UID: "baf50b51-6675-4e84-8793-40129e8abf91"). InnerVolumeSpecName "kube-api-access-vlvpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:58:43.398390 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:43.398283 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlvpl\" (UniqueName: \"kubernetes.io/projected/baf50b51-6675-4e84-8793-40129e8abf91-kube-api-access-vlvpl\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:58:44.076610 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:44.076581 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7852-140045906377648" Jul 23 00:58:44.076805 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:44.076612 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a780960eb46f1405eee7d81cb8d0de4655a70254c58a836ad8a48ffeb305c4d6" Jul 23 00:58:44.087425 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:58:44.087387 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7852-140045906377648\" is forbidden: User \"system:node:ip-10-0-131-121.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-121.ec2.internal' and this object" podUID="baf50b51-6675-4e84-8793-40129e8abf91" pod="opendatahub/test-curl-7852-140045906377648" Jul 23 00:58:45.007248 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:45.007211 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="baf50b51-6675-4e84-8793-40129e8abf91" path="/var/lib/kubelet/pods/baf50b51-6675-4e84-8793-40129e8abf91/volumes" Jul 23 00:58:47.009647 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:47.009612 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7852-140045906543648"] Jul 23 00:58:47.013612 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:47.010507 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="baf50b51-6675-4e84-8793-40129e8abf91" containerName="test-curl-7852-140045906377648" Jul 23 00:58:47.013612 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:47.010535 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="baf50b51-6675-4e84-8793-40129e8abf91" containerName="test-curl-7852-140045906377648" Jul 23 00:58:47.013612 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:47.010865 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="baf50b51-6675-4e84-8793-40129e8abf91" containerName="test-curl-7852-140045906377648" Jul 23 00:58:47.015702 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:47.015673 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7852-140045906543648" Jul 23 00:58:47.022076 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:47.020972 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-wjlfl\"" Jul 23 00:58:47.026043 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:47.026010 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7852-140045906543648"] Jul 23 00:58:47.035464 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:47.035428 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkllf\" (UniqueName: \"kubernetes.io/projected/21db7f1e-837a-4235-a1e4-5214a3989a03-kube-api-access-fkllf\") pod \"test-curl-7852-140045906543648\" (UID: \"21db7f1e-837a-4235-a1e4-5214a3989a03\") " pod="opendatahub/test-curl-7852-140045906543648" Jul 23 00:58:47.136895 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:47.136855 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkllf\" (UniqueName: \"kubernetes.io/projected/21db7f1e-837a-4235-a1e4-5214a3989a03-kube-api-access-fkllf\") pod \"test-curl-7852-140045906543648\" (UID: \"21db7f1e-837a-4235-a1e4-5214a3989a03\") " pod="opendatahub/test-curl-7852-140045906543648" Jul 23 00:58:47.145837 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:47.145807 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkllf\" (UniqueName: \"kubernetes.io/projected/21db7f1e-837a-4235-a1e4-5214a3989a03-kube-api-access-fkllf\") pod \"test-curl-7852-140045906543648\" (UID: \"21db7f1e-837a-4235-a1e4-5214a3989a03\") " pod="opendatahub/test-curl-7852-140045906543648" Jul 23 00:58:47.330077 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:47.330027 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7852-140045906543648" Jul 23 00:58:47.463705 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:47.463680 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7852-140045906543648"] Jul 23 00:58:47.466414 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:58:47.466386 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod21db7f1e_837a_4235_a1e4_5214a3989a03.slice/crio-f5bafcb12cf69ec7d9a78ee72b86eec9e2a34910fee7ec602a6e7c25501bf49c WatchSource:0}: Error finding container f5bafcb12cf69ec7d9a78ee72b86eec9e2a34910fee7ec602a6e7c25501bf49c: Status 404 returned error can't find the container with id f5bafcb12cf69ec7d9a78ee72b86eec9e2a34910fee7ec602a6e7c25501bf49c Jul 23 00:58:48.101502 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:48.101463 2562 generic.go:356] "Generic (PLEG): container finished" podID="21db7f1e-837a-4235-a1e4-5214a3989a03" containerID="b79da7c5f19aa59356e576d37a759f738780aa70f7d3d531ab5859e53e791aed" exitCode=0 Jul 23 00:58:48.101928 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:48.101548 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7852-140045906543648" event={"ID":"21db7f1e-837a-4235-a1e4-5214a3989a03","Type":"ContainerDied","Data":"b79da7c5f19aa59356e576d37a759f738780aa70f7d3d531ab5859e53e791aed"} Jul 23 00:58:48.101928 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:48.101587 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7852-140045906543648" event={"ID":"21db7f1e-837a-4235-a1e4-5214a3989a03","Type":"ContainerStarted","Data":"f5bafcb12cf69ec7d9a78ee72b86eec9e2a34910fee7ec602a6e7c25501bf49c"} Jul 23 00:58:49.246685 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:49.246662 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7852-140045906543648" Jul 23 00:58:49.270343 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:49.270301 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7852-140045906543648_21db7f1e-837a-4235-a1e4-5214a3989a03/test-curl-7852-140045906543648/0.log" Jul 23 00:58:49.300825 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:49.300787 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7852-140045906543648"] Jul 23 00:58:49.309198 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:49.309145 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7852-140045906543648"] Jul 23 00:58:49.359103 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:49.358998 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fkllf\" (UniqueName: \"kubernetes.io/projected/21db7f1e-837a-4235-a1e4-5214a3989a03-kube-api-access-fkllf\") pod \"21db7f1e-837a-4235-a1e4-5214a3989a03\" (UID: \"21db7f1e-837a-4235-a1e4-5214a3989a03\") " Jul 23 00:58:49.361168 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:49.361139 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21db7f1e-837a-4235-a1e4-5214a3989a03-kube-api-access-fkllf" (OuterVolumeSpecName: "kube-api-access-fkllf") pod "21db7f1e-837a-4235-a1e4-5214a3989a03" (UID: "21db7f1e-837a-4235-a1e4-5214a3989a03"). InnerVolumeSpecName "kube-api-access-fkllf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:58:49.433573 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:49.433532 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7852-140045906543648"] Jul 23 00:58:49.433945 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:49.433931 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21db7f1e-837a-4235-a1e4-5214a3989a03" containerName="test-curl-7852-140045906543648" Jul 23 00:58:49.434004 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:49.433946 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="21db7f1e-837a-4235-a1e4-5214a3989a03" containerName="test-curl-7852-140045906543648" Jul 23 00:58:49.434050 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:49.434032 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="21db7f1e-837a-4235-a1e4-5214a3989a03" containerName="test-curl-7852-140045906543648" Jul 23 00:58:49.437059 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:49.437042 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7852-140045906543648" Jul 23 00:58:49.446604 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:49.446578 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7852-140045906543648"] Jul 23 00:58:49.460612 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:49.460585 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkllf\" (UniqueName: \"kubernetes.io/projected/21db7f1e-837a-4235-a1e4-5214a3989a03-kube-api-access-fkllf\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:58:49.561788 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:49.561725 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7gqq\" (UniqueName: \"kubernetes.io/projected/72d29134-f05d-4a92-88ba-0731387062a4-kube-api-access-q7gqq\") pod \"test-curl-7852-140045906543648\" (UID: \"72d29134-f05d-4a92-88ba-0731387062a4\") " pod="opendatahub/test-curl-7852-140045906543648" Jul 23 00:58:49.663304 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:49.663214 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7gqq\" (UniqueName: \"kubernetes.io/projected/72d29134-f05d-4a92-88ba-0731387062a4-kube-api-access-q7gqq\") pod \"test-curl-7852-140045906543648\" (UID: \"72d29134-f05d-4a92-88ba-0731387062a4\") " pod="opendatahub/test-curl-7852-140045906543648" Jul 23 00:58:49.672842 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:49.672812 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7gqq\" (UniqueName: \"kubernetes.io/projected/72d29134-f05d-4a92-88ba-0731387062a4-kube-api-access-q7gqq\") pod \"test-curl-7852-140045906543648\" (UID: \"72d29134-f05d-4a92-88ba-0731387062a4\") " pod="opendatahub/test-curl-7852-140045906543648" Jul 23 00:58:49.747602 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:49.747562 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7852-140045906543648" Jul 23 00:58:49.917341 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:49.917313 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7852-140045906543648"] Jul 23 00:58:49.919100 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:58:49.919071 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod72d29134_f05d_4a92_88ba_0731387062a4.slice/crio-68d9fd7f9a74bf518e73e4f815ef2d115e00f794632a1875e8c112076743363f WatchSource:0}: Error finding container 68d9fd7f9a74bf518e73e4f815ef2d115e00f794632a1875e8c112076743363f: Status 404 returned error can't find the container with id 68d9fd7f9a74bf518e73e4f815ef2d115e00f794632a1875e8c112076743363f Jul 23 00:58:50.114323 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:50.114282 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7852-140045906543648" event={"ID":"72d29134-f05d-4a92-88ba-0731387062a4","Type":"ContainerStarted","Data":"68d9fd7f9a74bf518e73e4f815ef2d115e00f794632a1875e8c112076743363f"} Jul 23 00:58:50.115516 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:50.115499 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7852-140045906543648" Jul 23 00:58:50.115655 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:50.115543 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5bafcb12cf69ec7d9a78ee72b86eec9e2a34910fee7ec602a6e7c25501bf49c" Jul 23 00:58:50.128039 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:50.128001 2562 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7852-140045906543648" oldPodUID="21db7f1e-837a-4235-a1e4-5214a3989a03" podUID="72d29134-f05d-4a92-88ba-0731387062a4" Jul 23 00:58:51.007632 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:51.007597 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21db7f1e-837a-4235-a1e4-5214a3989a03" path="/var/lib/kubelet/pods/21db7f1e-837a-4235-a1e4-5214a3989a03/volumes" Jul 23 00:58:51.123456 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:51.123417 2562 generic.go:356] "Generic (PLEG): container finished" podID="72d29134-f05d-4a92-88ba-0731387062a4" containerID="9cb9b0fe2842de6e434808aceb343f4f914b60e421a1965d5e3f3567a77d3cc0" exitCode=0 Jul 23 00:58:51.123636 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:51.123508 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7852-140045906543648" event={"ID":"72d29134-f05d-4a92-88ba-0731387062a4","Type":"ContainerDied","Data":"9cb9b0fe2842de6e434808aceb343f4f914b60e421a1965d5e3f3567a77d3cc0"} Jul 23 00:58:52.263424 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:52.263397 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7852-140045906543648" Jul 23 00:58:52.286116 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:52.286080 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7852-140045906543648_72d29134-f05d-4a92-88ba-0731387062a4/test-curl-7852-140045906543648/0.log" Jul 23 00:58:52.310339 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:52.310301 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7852-140045906543648"] Jul 23 00:58:52.313824 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:52.313794 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7852-140045906543648"] Jul 23 00:58:52.396739 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:52.396645 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q7gqq\" (UniqueName: \"kubernetes.io/projected/72d29134-f05d-4a92-88ba-0731387062a4-kube-api-access-q7gqq\") pod \"72d29134-f05d-4a92-88ba-0731387062a4\" (UID: \"72d29134-f05d-4a92-88ba-0731387062a4\") " Jul 23 00:58:52.398779 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:52.398725 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72d29134-f05d-4a92-88ba-0731387062a4-kube-api-access-q7gqq" (OuterVolumeSpecName: "kube-api-access-q7gqq") pod "72d29134-f05d-4a92-88ba-0731387062a4" (UID: "72d29134-f05d-4a92-88ba-0731387062a4"). InnerVolumeSpecName "kube-api-access-q7gqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:58:52.498387 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:52.498344 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7gqq\" (UniqueName: \"kubernetes.io/projected/72d29134-f05d-4a92-88ba-0731387062a4-kube-api-access-q7gqq\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:58:53.007380 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:53.007343 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72d29134-f05d-4a92-88ba-0731387062a4" path="/var/lib/kubelet/pods/72d29134-f05d-4a92-88ba-0731387062a4/volumes" Jul 23 00:58:53.136007 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:53.135976 2562 scope.go:117] "RemoveContainer" containerID="9cb9b0fe2842de6e434808aceb343f4f914b60e421a1965d5e3f3567a77d3cc0" Jul 23 00:58:53.136200 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:58:53.135983 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7852-140045906543648" Jul 23 00:59:07.226450 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:07.226410 2562 scope.go:117] "RemoveContainer" containerID="8113b33143dedcbc72e7e48ebfdac464ff1eebe55d8e05648b9644b4d365da09" Jul 23 00:59:20.415847 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:20.415807 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d29ed6-xfq62"] Jul 23 00:59:20.416353 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:20.416245 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72d29134-f05d-4a92-88ba-0731387062a4" containerName="test-curl-7852-140045906543648" Jul 23 00:59:20.416353 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:20.416258 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="72d29134-f05d-4a92-88ba-0731387062a4" containerName="test-curl-7852-140045906543648" Jul 23 00:59:20.416353 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:20.416321 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="72d29134-f05d-4a92-88ba-0731387062a4" containerName="test-curl-7852-140045906543648" Jul 23 00:59:20.419586 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:20.419565 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d29ed6-xfq62" Jul 23 00:59:20.422612 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:20.422528 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w8tk2\"" Jul 23 00:59:20.426487 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:20.426448 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d29ed6-xfq62"] Jul 23 00:59:20.466108 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:20.466069 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/212c307c-9ece-4cd8-85ac-da23806fc892-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-d29ed6-xfq62\" (UID: \"212c307c-9ece-4cd8-85ac-da23806fc892\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d29ed6-xfq62" Jul 23 00:59:20.567038 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:20.566990 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/212c307c-9ece-4cd8-85ac-da23806fc892-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-d29ed6-xfq62\" (UID: \"212c307c-9ece-4cd8-85ac-da23806fc892\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d29ed6-xfq62" Jul 23 00:59:20.567667 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:20.567646 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/212c307c-9ece-4cd8-85ac-da23806fc892-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-d29ed6-xfq62\" (UID: \"212c307c-9ece-4cd8-85ac-da23806fc892\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d29ed6-xfq62" Jul 23 00:59:20.732009 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:20.731915 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d29ed6-xfq62" Jul 23 00:59:20.881594 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:20.881559 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d29ed6-xfq62"] Jul 23 00:59:20.881743 ip-10-0-131-121 kubenswrapper[2562]: W0723 00:59:20.881683 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod212c307c_9ece_4cd8_85ac_da23806fc892.slice/crio-505202513fe071c2514e95620eba9b2950810aae0aabaa7aab9947168037e815 WatchSource:0}: Error finding container 505202513fe071c2514e95620eba9b2950810aae0aabaa7aab9947168037e815: Status 404 returned error can't find the container with id 505202513fe071c2514e95620eba9b2950810aae0aabaa7aab9947168037e815 Jul 23 00:59:21.297459 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:21.297417 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d29ed6-xfq62" event={"ID":"212c307c-9ece-4cd8-85ac-da23806fc892","Type":"ContainerStarted","Data":"521d27578a4c6a90a1d0eff3330b4d53799f47106bc6c8d8d8b900d8210b0da6"} Jul 23 00:59:21.297631 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:21.297489 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d29ed6-xfq62" event={"ID":"212c307c-9ece-4cd8-85ac-da23806fc892","Type":"ContainerStarted","Data":"505202513fe071c2514e95620eba9b2950810aae0aabaa7aab9947168037e815"} Jul 23 00:59:21.330902 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:21.330848 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d29ed6-xfq62" podStartSLOduration=1.330829343 podStartE2EDuration="1.330829343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:59:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:59:21.328082605 +0000 UTC m=+3075.123589477" watchObservedRunningTime="2026-07-23 00:59:21.330829343 +0000 UTC m=+3075.126336199" Jul 23 00:59:22.304291 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:22.304254 2562 generic.go:356] "Generic (PLEG): container finished" podID="212c307c-9ece-4cd8-85ac-da23806fc892" containerID="521d27578a4c6a90a1d0eff3330b4d53799f47106bc6c8d8d8b900d8210b0da6" exitCode=0 Jul 23 00:59:22.304732 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:22.304342 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d29ed6-xfq62" event={"ID":"212c307c-9ece-4cd8-85ac-da23806fc892","Type":"ContainerDied","Data":"521d27578a4c6a90a1d0eff3330b4d53799f47106bc6c8d8d8b900d8210b0da6"} Jul 23 00:59:23.449649 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:23.449618 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d29ed6-xfq62" Jul 23 00:59:23.495696 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:23.495635 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/212c307c-9ece-4cd8-85ac-da23806fc892-maas-api-service-ca\") pod \"212c307c-9ece-4cd8-85ac-da23806fc892\" (UID: \"212c307c-9ece-4cd8-85ac-da23806fc892\") " Jul 23 00:59:23.496096 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:23.496068 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/212c307c-9ece-4cd8-85ac-da23806fc892-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "212c307c-9ece-4cd8-85ac-da23806fc892" (UID: "212c307c-9ece-4cd8-85ac-da23806fc892"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:59:23.597621 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:23.597525 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/212c307c-9ece-4cd8-85ac-da23806fc892-maas-api-service-ca\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:59:24.318037 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:24.318006 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d29ed6-xfq62" Jul 23 00:59:24.318224 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:24.318040 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d29ed6-xfq62" event={"ID":"212c307c-9ece-4cd8-85ac-da23806fc892","Type":"ContainerDied","Data":"505202513fe071c2514e95620eba9b2950810aae0aabaa7aab9947168037e815"} Jul 23 00:59:24.318224 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:24.318090 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="505202513fe071c2514e95620eba9b2950810aae0aabaa7aab9947168037e815" Jul 23 00:59:35.506078 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:35.506036 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d29ed6-xfq62"] Jul 23 00:59:35.510740 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:35.510707 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d29ed6-xfq62"] Jul 23 00:59:37.008103 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:37.008066 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="212c307c-9ece-4cd8-85ac-da23806fc892" path="/var/lib/kubelet/pods/212c307c-9ece-4cd8-85ac-da23806fc892/volumes" Jul 23 00:59:42.141815 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:42.141751 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz"] Jul 23 00:59:42.142258 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:42.141983 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz" podUID="4afd8bba-2465-4d15-a7b8-4550cfd379f3" containerName="istio-proxy" containerID="cri-o://3eab1e1cb3ff3ed8ea83b6638e6b2294ed99538bb543c64aef6d443e71005bf0" gracePeriod=30 Jul 23 00:59:47.404434 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:47.404357 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz" Jul 23 00:59:47.452722 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:47.452684 2562 generic.go:356] "Generic (PLEG): container finished" podID="4afd8bba-2465-4d15-a7b8-4550cfd379f3" containerID="3eab1e1cb3ff3ed8ea83b6638e6b2294ed99538bb543c64aef6d443e71005bf0" exitCode=0 Jul 23 00:59:47.452906 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:47.452732 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz" Jul 23 00:59:47.452906 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:47.452751 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz" event={"ID":"4afd8bba-2465-4d15-a7b8-4550cfd379f3","Type":"ContainerDied","Data":"3eab1e1cb3ff3ed8ea83b6638e6b2294ed99538bb543c64aef6d443e71005bf0"} Jul 23 00:59:47.452906 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:47.452812 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz" event={"ID":"4afd8bba-2465-4d15-a7b8-4550cfd379f3","Type":"ContainerDied","Data":"1fef00766007da37b43b197591a2dab05816ebf6c775e8ef86a6f69eb660e95a"} Jul 23 00:59:47.452906 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:47.452833 2562 scope.go:117] "RemoveContainer" containerID="3eab1e1cb3ff3ed8ea83b6638e6b2294ed99538bb543c64aef6d443e71005bf0" Jul 23 00:59:47.462181 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:47.462156 2562 scope.go:117] "RemoveContainer" containerID="9618eced548337b615a501148a663baf4232bf90f11aa2d3b64e6d4cbf4da837" Jul 23 00:59:47.470983 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:47.470954 2562 scope.go:117] "RemoveContainer" containerID="3eab1e1cb3ff3ed8ea83b6638e6b2294ed99538bb543c64aef6d443e71005bf0" Jul 23 00:59:47.471319 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:59:47.471297 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3eab1e1cb3ff3ed8ea83b6638e6b2294ed99538bb543c64aef6d443e71005bf0\": container with ID starting with 3eab1e1cb3ff3ed8ea83b6638e6b2294ed99538bb543c64aef6d443e71005bf0 not found: ID does not exist" containerID="3eab1e1cb3ff3ed8ea83b6638e6b2294ed99538bb543c64aef6d443e71005bf0" Jul 23 00:59:47.471369 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:47.471330 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3eab1e1cb3ff3ed8ea83b6638e6b2294ed99538bb543c64aef6d443e71005bf0"} err="failed to get container status \"3eab1e1cb3ff3ed8ea83b6638e6b2294ed99538bb543c64aef6d443e71005bf0\": rpc error: code = NotFound desc = could not find container \"3eab1e1cb3ff3ed8ea83b6638e6b2294ed99538bb543c64aef6d443e71005bf0\": container with ID starting with 3eab1e1cb3ff3ed8ea83b6638e6b2294ed99538bb543c64aef6d443e71005bf0 not found: ID does not exist" Jul 23 00:59:47.471369 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:47.471350 2562 scope.go:117] "RemoveContainer" containerID="9618eced548337b615a501148a663baf4232bf90f11aa2d3b64e6d4cbf4da837" Jul 23 00:59:47.471628 ip-10-0-131-121 kubenswrapper[2562]: E0723 00:59:47.471609 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9618eced548337b615a501148a663baf4232bf90f11aa2d3b64e6d4cbf4da837\": container with ID starting with 9618eced548337b615a501148a663baf4232bf90f11aa2d3b64e6d4cbf4da837 not found: ID does not exist" containerID="9618eced548337b615a501148a663baf4232bf90f11aa2d3b64e6d4cbf4da837" Jul 23 00:59:47.471677 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:47.471635 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9618eced548337b615a501148a663baf4232bf90f11aa2d3b64e6d4cbf4da837"} err="failed to get container status \"9618eced548337b615a501148a663baf4232bf90f11aa2d3b64e6d4cbf4da837\": rpc error: code = NotFound desc = could not find container \"9618eced548337b615a501148a663baf4232bf90f11aa2d3b64e6d4cbf4da837\": container with ID starting with 9618eced548337b615a501148a663baf4232bf90f11aa2d3b64e6d4cbf4da837 not found: ID does not exist" Jul 23 00:59:47.522672 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:47.522630 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4afd8bba-2465-4d15-a7b8-4550cfd379f3-istio-data\") pod \"4afd8bba-2465-4d15-a7b8-4550cfd379f3\" (UID: \"4afd8bba-2465-4d15-a7b8-4550cfd379f3\") " Jul 23 00:59:47.522905 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:47.522707 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4afd8bba-2465-4d15-a7b8-4550cfd379f3-istio-podinfo\") pod \"4afd8bba-2465-4d15-a7b8-4550cfd379f3\" (UID: \"4afd8bba-2465-4d15-a7b8-4550cfd379f3\") " Jul 23 00:59:47.522905 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:47.522747 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4afd8bba-2465-4d15-a7b8-4550cfd379f3-istio-token\") pod \"4afd8bba-2465-4d15-a7b8-4550cfd379f3\" (UID: \"4afd8bba-2465-4d15-a7b8-4550cfd379f3\") " Jul 23 00:59:47.522905 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:47.522802 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4afd8bba-2465-4d15-a7b8-4550cfd379f3-credential-socket\") pod \"4afd8bba-2465-4d15-a7b8-4550cfd379f3\" (UID: \"4afd8bba-2465-4d15-a7b8-4550cfd379f3\") " Jul 23 00:59:47.523096 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:47.522964 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4afd8bba-2465-4d15-a7b8-4550cfd379f3-istio-envoy\") pod \"4afd8bba-2465-4d15-a7b8-4550cfd379f3\" (UID: \"4afd8bba-2465-4d15-a7b8-4550cfd379f3\") " Jul 23 00:59:47.523096 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:47.522994 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4afd8bba-2465-4d15-a7b8-4550cfd379f3-workload-socket\") pod \"4afd8bba-2465-4d15-a7b8-4550cfd379f3\" (UID: \"4afd8bba-2465-4d15-a7b8-4550cfd379f3\") " Jul 23 00:59:47.523096 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:47.523019 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4afd8bba-2465-4d15-a7b8-4550cfd379f3-istiod-ca-cert\") pod \"4afd8bba-2465-4d15-a7b8-4550cfd379f3\" (UID: \"4afd8bba-2465-4d15-a7b8-4550cfd379f3\") " Jul 23 00:59:47.523096 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:47.523061 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4afd8bba-2465-4d15-a7b8-4550cfd379f3-workload-certs\") pod \"4afd8bba-2465-4d15-a7b8-4550cfd379f3\" (UID: \"4afd8bba-2465-4d15-a7b8-4550cfd379f3\") " Jul 23 00:59:47.523096 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:47.523090 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d8sw5\" (UniqueName: \"kubernetes.io/projected/4afd8bba-2465-4d15-a7b8-4550cfd379f3-kube-api-access-d8sw5\") pod \"4afd8bba-2465-4d15-a7b8-4550cfd379f3\" (UID: \"4afd8bba-2465-4d15-a7b8-4550cfd379f3\") " Jul 23 00:59:47.523352 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:47.523108 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4afd8bba-2465-4d15-a7b8-4550cfd379f3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4afd8bba-2465-4d15-a7b8-4550cfd379f3" (UID: "4afd8bba-2465-4d15-a7b8-4550cfd379f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:59:47.523352 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:47.523271 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4afd8bba-2465-4d15-a7b8-4550cfd379f3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4afd8bba-2465-4d15-a7b8-4550cfd379f3" (UID: "4afd8bba-2465-4d15-a7b8-4550cfd379f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:59:47.523471 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:47.523444 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4afd8bba-2465-4d15-a7b8-4550cfd379f3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4afd8bba-2465-4d15-a7b8-4550cfd379f3" (UID: "4afd8bba-2465-4d15-a7b8-4550cfd379f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:59:47.523524 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:47.523490 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4afd8bba-2465-4d15-a7b8-4550cfd379f3-credential-socket\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:59:47.523524 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:47.523512 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4afd8bba-2465-4d15-a7b8-4550cfd379f3-workload-socket\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:59:47.523723 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:47.523683 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4afd8bba-2465-4d15-a7b8-4550cfd379f3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4afd8bba-2465-4d15-a7b8-4550cfd379f3" (UID: "4afd8bba-2465-4d15-a7b8-4550cfd379f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:59:47.523823 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:47.523735 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4afd8bba-2465-4d15-a7b8-4550cfd379f3-istio-data" (OuterVolumeSpecName: "istio-data") pod "4afd8bba-2465-4d15-a7b8-4550cfd379f3" (UID: "4afd8bba-2465-4d15-a7b8-4550cfd379f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:59:47.525209 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:47.525187 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4afd8bba-2465-4d15-a7b8-4550cfd379f3-kube-api-access-d8sw5" (OuterVolumeSpecName: "kube-api-access-d8sw5") pod "4afd8bba-2465-4d15-a7b8-4550cfd379f3" (UID: "4afd8bba-2465-4d15-a7b8-4550cfd379f3"). InnerVolumeSpecName "kube-api-access-d8sw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:59:47.525310 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:47.525231 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4afd8bba-2465-4d15-a7b8-4550cfd379f3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4afd8bba-2465-4d15-a7b8-4550cfd379f3" (UID: "4afd8bba-2465-4d15-a7b8-4550cfd379f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:59:47.525408 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:47.525392 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4afd8bba-2465-4d15-a7b8-4550cfd379f3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4afd8bba-2465-4d15-a7b8-4550cfd379f3" (UID: "4afd8bba-2465-4d15-a7b8-4550cfd379f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 00:59:47.525473 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:47.525413 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4afd8bba-2465-4d15-a7b8-4550cfd379f3-istio-token" (OuterVolumeSpecName: "istio-token") pod "4afd8bba-2465-4d15-a7b8-4550cfd379f3" (UID: "4afd8bba-2465-4d15-a7b8-4550cfd379f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:59:47.624416 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:47.624378 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4afd8bba-2465-4d15-a7b8-4550cfd379f3-istio-envoy\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:59:47.624416 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:47.624411 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4afd8bba-2465-4d15-a7b8-4550cfd379f3-istiod-ca-cert\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:59:47.624416 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:47.624421 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4afd8bba-2465-4d15-a7b8-4550cfd379f3-workload-certs\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:59:47.624672 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:47.624430 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8sw5\" (UniqueName: \"kubernetes.io/projected/4afd8bba-2465-4d15-a7b8-4550cfd379f3-kube-api-access-d8sw5\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:59:47.624672 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:47.624441 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4afd8bba-2465-4d15-a7b8-4550cfd379f3-istio-data\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:59:47.624672 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:47.624449 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4afd8bba-2465-4d15-a7b8-4550cfd379f3-istio-podinfo\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:59:47.624672 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:47.624457 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4afd8bba-2465-4d15-a7b8-4550cfd379f3-istio-token\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 23 00:59:47.795217 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:47.795164 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz"] Jul 23 00:59:47.799873 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:47.799840 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-d29ed6-openshift-default-75d9f674c-8djxz"] Jul 23 00:59:49.007576 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:49.007541 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4afd8bba-2465-4d15-a7b8-4550cfd379f3" path="/var/lib/kubelet/pods/4afd8bba-2465-4d15-a7b8-4550cfd379f3/volumes" Jul 23 00:59:49.058725 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:49.058701 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f65bfdcb7-k8wsd_48c0a1e1-c11f-405e-baeb-98d98897dd11/copy-manifests/0.log" Jul 23 00:59:49.073367 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:49.073328 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f65bfdcb7-k8wsd_48c0a1e1-c11f-405e-baeb-98d98897dd11/manager/0.log" Jul 23 00:59:49.684825 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:49.684794 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29746110-jbxnb_26fd18d8-a27c-4ad6-8e87-7dfe920bd797/cleanup/0.log" Jul 23 00:59:49.804886 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:49.804853 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29746125-s4rhn_60a410ac-a77e-47e3-8b3c-31bceacecc1d/cleanup/0.log" Jul 23 00:59:49.924606 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:49.924578 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-pvbhb_c5796969-96a2-4b83-a70e-ebdf37f2b36e/postgres/0.log" Jul 23 00:59:50.753597 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:50.753568 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hw9f7_cc5b0c3a-51dc-4e24-a6de-140c0ecb8f24/util/0.log" Jul 23 00:59:50.767881 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:50.767858 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hw9f7_cc5b0c3a-51dc-4e24-a6de-140c0ecb8f24/pull/0.log" Jul 23 00:59:50.788360 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:50.788336 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hw9f7_cc5b0c3a-51dc-4e24-a6de-140c0ecb8f24/extract/0.log" Jul 23 00:59:50.951388 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:50.951355 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pxctb_a19a892d-c6f3-451e-b60b-824e30993eae/util/0.log" Jul 23 00:59:50.966165 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:50.966139 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pxctb_a19a892d-c6f3-451e-b60b-824e30993eae/pull/0.log" Jul 23 00:59:50.984505 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:50.984479 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pxctb_a19a892d-c6f3-451e-b60b-824e30993eae/extract/0.log" Jul 23 00:59:51.113459 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:51.113431 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nn4k6_2e929649-eea1-4da3-ab01-a77d5c4a0d34/util/0.log" Jul 23 00:59:51.127487 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:51.127464 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nn4k6_2e929649-eea1-4da3-ab01-a77d5c4a0d34/pull/0.log" Jul 23 00:59:51.134394 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:51.134372 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nn4k6_2e929649-eea1-4da3-ab01-a77d5c4a0d34/extract/0.log" Jul 23 00:59:51.251786 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:51.251733 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nm6kj_3299b5b8-cd31-4b1f-9f76-dcbf5b1f5337/util/0.log" Jul 23 00:59:51.258564 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:51.258540 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nm6kj_3299b5b8-cd31-4b1f-9f76-dcbf5b1f5337/pull/0.log" Jul 23 00:59:51.265658 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:51.265634 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nm6kj_3299b5b8-cd31-4b1f-9f76-dcbf5b1f5337/extract/0.log" Jul 23 00:59:52.554730 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:52.554701 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678bfjwm_a5b4d97f-58c7-4032-85f5-8c31e65c8947/istio-proxy/0.log" Jul 23 00:59:52.691091 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:52.691062 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-t62mx_557397d5-d277-4ee8-a5bd-bbc4fec09acb/discovery/0.log" Jul 23 00:59:53.341739 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:53.341705 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-glvxf_baedce3c-d14a-49ac-b183-8468e48c8d89/payload-processing/0.log" Jul 23 00:59:53.476365 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:53.476335 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-76974999cb-mggnm_c0ba4e0d-2b3e-4a20-8f3e-1296830be034/router/0.log" Jul 23 00:59:54.078817 ip-10-0-131-121 kubenswrapper[2562]: I0723 00:59:54.078790 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-b7w99_5635ea5d-e3cc-46fe-b8dd-e498b18018fe/main/0.log" Jul 23 01:00:03.466105 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:03.466050 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-jlsd6_5f272157-a81d-439c-86ed-511e11146698/global-pull-secret-syncer/0.log" Jul 23 01:00:03.550447 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:03.550402 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-fhbjd_19f5db96-5df5-4b0c-8cc1-89d3f2719ed2/konnectivity-agent/0.log" Jul 23 01:00:03.609019 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:03.608970 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-121.ec2.internal_19e11d045e1035a525da0259bed898ce/haproxy/0.log" Jul 23 01:00:08.288106 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:08.288059 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hw9f7_cc5b0c3a-51dc-4e24-a6de-140c0ecb8f24/extract/0.log" Jul 23 01:00:08.312726 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:08.312694 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hw9f7_cc5b0c3a-51dc-4e24-a6de-140c0ecb8f24/util/0.log" Jul 23 01:00:08.338830 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:08.338778 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hw9f7_cc5b0c3a-51dc-4e24-a6de-140c0ecb8f24/pull/0.log" Jul 23 01:00:08.368079 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:08.368046 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pxctb_a19a892d-c6f3-451e-b60b-824e30993eae/extract/0.log" Jul 23 01:00:08.392159 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:08.392128 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pxctb_a19a892d-c6f3-451e-b60b-824e30993eae/util/0.log" Jul 23 01:00:08.419620 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:08.419591 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pxctb_a19a892d-c6f3-451e-b60b-824e30993eae/pull/0.log" Jul 23 01:00:08.449610 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:08.449553 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nn4k6_2e929649-eea1-4da3-ab01-a77d5c4a0d34/extract/0.log" Jul 23 01:00:08.475414 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:08.475380 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nn4k6_2e929649-eea1-4da3-ab01-a77d5c4a0d34/util/0.log" Jul 23 01:00:08.510321 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:08.510290 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nn4k6_2e929649-eea1-4da3-ab01-a77d5c4a0d34/pull/0.log" Jul 23 01:00:08.541211 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:08.541131 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nm6kj_3299b5b8-cd31-4b1f-9f76-dcbf5b1f5337/extract/0.log" Jul 23 01:00:08.570993 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:08.570960 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nm6kj_3299b5b8-cd31-4b1f-9f76-dcbf5b1f5337/util/0.log" Jul 23 01:00:08.599327 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:08.599301 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1nm6kj_3299b5b8-cd31-4b1f-9f76-dcbf5b1f5337/pull/0.log" Jul 23 01:00:10.982917 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:10.982886 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_60761a85-20d1-4175-a35b-34bd82651f37/alertmanager/0.log" Jul 23 01:00:11.010410 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:11.010384 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_60761a85-20d1-4175-a35b-34bd82651f37/config-reloader/0.log" Jul 23 01:00:11.039296 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:11.039256 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_60761a85-20d1-4175-a35b-34bd82651f37/kube-rbac-proxy-web/0.log" Jul 23 01:00:11.070084 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:11.070033 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_60761a85-20d1-4175-a35b-34bd82651f37/kube-rbac-proxy/0.log" Jul 23 01:00:11.100716 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:11.100685 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_60761a85-20d1-4175-a35b-34bd82651f37/kube-rbac-proxy-metric/0.log" Jul 23 01:00:11.132354 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:11.132327 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_60761a85-20d1-4175-a35b-34bd82651f37/prom-label-proxy/0.log" Jul 23 01:00:11.158344 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:11.158318 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_60761a85-20d1-4175-a35b-34bd82651f37/init-config-reloader/0.log" Jul 23 01:00:11.470615 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:11.470584 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-9947f897b-pt65r_4b64b4a7-4872-4de0-ae43-039812f2cb1f/metrics-server/0.log" Jul 23 01:00:11.555731 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:11.555698 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-7b7b7bc8bf-q6j98_cc3da55a-09e5-4b6b-b2b6-b973078b728a/monitoring-plugin/0.log" Jul 23 01:00:11.754286 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:11.754203 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dscmh_34ac04c8-8aed-457a-b3f0-5235d1f2c421/node-exporter/0.log" Jul 23 01:00:11.782609 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:11.782574 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dscmh_34ac04c8-8aed-457a-b3f0-5235d1f2c421/kube-rbac-proxy/0.log" Jul 23 01:00:11.805525 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:11.805496 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dscmh_34ac04c8-8aed-457a-b3f0-5235d1f2c421/init-textfile/0.log" Jul 23 01:00:12.214229 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:12.214196 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-96859b58f-2d4mn_fda29169-d2fd-464b-87a6-ff7ce5dba70d/telemeter-client/0.log" Jul 23 01:00:12.253112 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:12.253078 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-96859b58f-2d4mn_fda29169-d2fd-464b-87a6-ff7ce5dba70d/reload/0.log" Jul 23 01:00:12.286033 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:12.285980 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-96859b58f-2d4mn_fda29169-d2fd-464b-87a6-ff7ce5dba70d/kube-rbac-proxy/0.log" Jul 23 01:00:12.318858 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:12.318830 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-94f4d8999-2kqks_a54ddaab-286b-4d31-ad83-8f59cdb42c9a/thanos-query/0.log" Jul 23 01:00:12.347004 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:12.346965 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-94f4d8999-2kqks_a54ddaab-286b-4d31-ad83-8f59cdb42c9a/kube-rbac-proxy-web/0.log" Jul 23 01:00:12.371835 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:12.371800 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-94f4d8999-2kqks_a54ddaab-286b-4d31-ad83-8f59cdb42c9a/kube-rbac-proxy/0.log" Jul 23 01:00:12.395830 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:12.395783 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-94f4d8999-2kqks_a54ddaab-286b-4d31-ad83-8f59cdb42c9a/prom-label-proxy/0.log" Jul 23 01:00:12.420428 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:12.420395 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-94f4d8999-2kqks_a54ddaab-286b-4d31-ad83-8f59cdb42c9a/kube-rbac-proxy-rules/0.log" Jul 23 01:00:12.458660 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:12.458618 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-94f4d8999-2kqks_a54ddaab-286b-4d31-ad83-8f59cdb42c9a/kube-rbac-proxy-metrics/0.log" Jul 23 01:00:13.502935 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:13.502897 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mtxqj/perf-node-gather-daemonset-sd5lh"] Jul 23 01:00:13.503359 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:13.503292 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4afd8bba-2465-4d15-a7b8-4550cfd379f3" containerName="istio-proxy" Jul 23 01:00:13.503359 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:13.503306 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4afd8bba-2465-4d15-a7b8-4550cfd379f3" containerName="istio-proxy" Jul 23 01:00:13.503359 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:13.503316 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4afd8bba-2465-4d15-a7b8-4550cfd379f3" containerName="istio-proxy" Jul 23 01:00:13.503359 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:13.503322 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4afd8bba-2465-4d15-a7b8-4550cfd379f3" containerName="istio-proxy" Jul 23 01:00:13.503359 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:13.503338 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="212c307c-9ece-4cd8-85ac-da23806fc892" containerName="revoke-keys" Jul 23 01:00:13.503359 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:13.503343 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="212c307c-9ece-4cd8-85ac-da23806fc892" containerName="revoke-keys" Jul 23 01:00:13.503550 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:13.503401 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="4afd8bba-2465-4d15-a7b8-4550cfd379f3" containerName="istio-proxy" Jul 23 01:00:13.503550 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:13.503409 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="212c307c-9ece-4cd8-85ac-da23806fc892" containerName="revoke-keys" Jul 23 01:00:13.503550 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:13.503415 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="4afd8bba-2465-4d15-a7b8-4550cfd379f3" containerName="istio-proxy" Jul 23 01:00:13.508317 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:13.508291 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mtxqj/perf-node-gather-daemonset-sd5lh" Jul 23 01:00:13.511033 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:13.511006 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mtxqj\"/\"default-dockercfg-t6rqd\"" Jul 23 01:00:13.512256 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:13.512239 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mtxqj\"/\"openshift-service-ca.crt\"" Jul 23 01:00:13.512328 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:13.512250 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mtxqj\"/\"kube-root-ca.crt\"" Jul 23 01:00:13.518190 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:13.518161 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mtxqj/perf-node-gather-daemonset-sd5lh"] Jul 23 01:00:13.567039 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:13.566992 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c6d82b83-2f52-4c43-8608-ca44295c4255-lib-modules\") pod \"perf-node-gather-daemonset-sd5lh\" (UID: \"c6d82b83-2f52-4c43-8608-ca44295c4255\") " pod="openshift-must-gather-mtxqj/perf-node-gather-daemonset-sd5lh" Jul 23 01:00:13.567039 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:13.567046 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c6d82b83-2f52-4c43-8608-ca44295c4255-proc\") pod \"perf-node-gather-daemonset-sd5lh\" (UID: \"c6d82b83-2f52-4c43-8608-ca44295c4255\") " pod="openshift-must-gather-mtxqj/perf-node-gather-daemonset-sd5lh" Jul 23 01:00:13.567271 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:13.567071 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkvkb\" (UniqueName: \"kubernetes.io/projected/c6d82b83-2f52-4c43-8608-ca44295c4255-kube-api-access-hkvkb\") pod \"perf-node-gather-daemonset-sd5lh\" (UID: \"c6d82b83-2f52-4c43-8608-ca44295c4255\") " pod="openshift-must-gather-mtxqj/perf-node-gather-daemonset-sd5lh" Jul 23 01:00:13.567271 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:13.567113 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c6d82b83-2f52-4c43-8608-ca44295c4255-podres\") pod \"perf-node-gather-daemonset-sd5lh\" (UID: \"c6d82b83-2f52-4c43-8608-ca44295c4255\") " pod="openshift-must-gather-mtxqj/perf-node-gather-daemonset-sd5lh" Jul 23 01:00:13.567271 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:13.567142 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c6d82b83-2f52-4c43-8608-ca44295c4255-sys\") pod \"perf-node-gather-daemonset-sd5lh\" (UID: \"c6d82b83-2f52-4c43-8608-ca44295c4255\") " pod="openshift-must-gather-mtxqj/perf-node-gather-daemonset-sd5lh" Jul 23 01:00:13.668490 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:13.668443 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c6d82b83-2f52-4c43-8608-ca44295c4255-podres\") pod \"perf-node-gather-daemonset-sd5lh\" (UID: \"c6d82b83-2f52-4c43-8608-ca44295c4255\") " pod="openshift-must-gather-mtxqj/perf-node-gather-daemonset-sd5lh" Jul 23 01:00:13.668684 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:13.668506 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c6d82b83-2f52-4c43-8608-ca44295c4255-sys\") pod \"perf-node-gather-daemonset-sd5lh\" (UID: \"c6d82b83-2f52-4c43-8608-ca44295c4255\") " pod="openshift-must-gather-mtxqj/perf-node-gather-daemonset-sd5lh" Jul 23 01:00:13.668684 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:13.668590 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c6d82b83-2f52-4c43-8608-ca44295c4255-lib-modules\") pod \"perf-node-gather-daemonset-sd5lh\" (UID: \"c6d82b83-2f52-4c43-8608-ca44295c4255\") " pod="openshift-must-gather-mtxqj/perf-node-gather-daemonset-sd5lh" Jul 23 01:00:13.668684 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:13.668629 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c6d82b83-2f52-4c43-8608-ca44295c4255-proc\") pod \"perf-node-gather-daemonset-sd5lh\" (UID: \"c6d82b83-2f52-4c43-8608-ca44295c4255\") " pod="openshift-must-gather-mtxqj/perf-node-gather-daemonset-sd5lh" Jul 23 01:00:13.668684 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:13.668644 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c6d82b83-2f52-4c43-8608-ca44295c4255-podres\") pod \"perf-node-gather-daemonset-sd5lh\" (UID: \"c6d82b83-2f52-4c43-8608-ca44295c4255\") " pod="openshift-must-gather-mtxqj/perf-node-gather-daemonset-sd5lh" Jul 23 01:00:13.668684 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:13.668660 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkvkb\" (UniqueName: \"kubernetes.io/projected/c6d82b83-2f52-4c43-8608-ca44295c4255-kube-api-access-hkvkb\") pod \"perf-node-gather-daemonset-sd5lh\" (UID: \"c6d82b83-2f52-4c43-8608-ca44295c4255\") " pod="openshift-must-gather-mtxqj/perf-node-gather-daemonset-sd5lh" Jul 23 01:00:13.668684 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:13.668662 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c6d82b83-2f52-4c43-8608-ca44295c4255-sys\") pod \"perf-node-gather-daemonset-sd5lh\" (UID: \"c6d82b83-2f52-4c43-8608-ca44295c4255\") " pod="openshift-must-gather-mtxqj/perf-node-gather-daemonset-sd5lh" Jul 23 01:00:13.668985 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:13.668712 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c6d82b83-2f52-4c43-8608-ca44295c4255-proc\") pod \"perf-node-gather-daemonset-sd5lh\" (UID: \"c6d82b83-2f52-4c43-8608-ca44295c4255\") " pod="openshift-must-gather-mtxqj/perf-node-gather-daemonset-sd5lh" Jul 23 01:00:13.668985 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:13.668823 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c6d82b83-2f52-4c43-8608-ca44295c4255-lib-modules\") pod \"perf-node-gather-daemonset-sd5lh\" (UID: \"c6d82b83-2f52-4c43-8608-ca44295c4255\") " pod="openshift-must-gather-mtxqj/perf-node-gather-daemonset-sd5lh" Jul 23 01:00:13.677966 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:13.677934 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkvkb\" (UniqueName: \"kubernetes.io/projected/c6d82b83-2f52-4c43-8608-ca44295c4255-kube-api-access-hkvkb\") pod \"perf-node-gather-daemonset-sd5lh\" (UID: \"c6d82b83-2f52-4c43-8608-ca44295c4255\") " pod="openshift-must-gather-mtxqj/perf-node-gather-daemonset-sd5lh" Jul 23 01:00:13.820548 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:13.820505 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mtxqj/perf-node-gather-daemonset-sd5lh" Jul 23 01:00:13.961656 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:13.961625 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mtxqj/perf-node-gather-daemonset-sd5lh"] Jul 23 01:00:13.963941 ip-10-0-131-121 kubenswrapper[2562]: W0723 01:00:13.963901 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc6d82b83_2f52_4c43_8608_ca44295c4255.slice/crio-9ca72b4b206e61729ba389e62c47009d7bb94f4f6193c4b0181381bfeccb0463 WatchSource:0}: Error finding container 9ca72b4b206e61729ba389e62c47009d7bb94f4f6193c4b0181381bfeccb0463: Status 404 returned error can't find the container with id 9ca72b4b206e61729ba389e62c47009d7bb94f4f6193c4b0181381bfeccb0463 Jul 23 01:00:14.618552 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:14.618515 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mtxqj/perf-node-gather-daemonset-sd5lh" event={"ID":"c6d82b83-2f52-4c43-8608-ca44295c4255","Type":"ContainerStarted","Data":"731c9821b2c183d444bb2c69134badf2b92aea47c98c673e41b80141744d2142"} Jul 23 01:00:14.619077 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:14.618576 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mtxqj/perf-node-gather-daemonset-sd5lh" event={"ID":"c6d82b83-2f52-4c43-8608-ca44295c4255","Type":"ContainerStarted","Data":"9ca72b4b206e61729ba389e62c47009d7bb94f4f6193c4b0181381bfeccb0463"} Jul 23 01:00:14.619077 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:14.618636 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mtxqj/perf-node-gather-daemonset-sd5lh" Jul 23 01:00:14.637270 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:14.637215 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mtxqj/perf-node-gather-daemonset-sd5lh" podStartSLOduration=1.637196324 podStartE2EDuration="1.637196324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 01:00:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:00:14.635886919 +0000 UTC m=+3128.431393777" watchObservedRunningTime="2026-07-23 01:00:14.637196324 +0000 UTC m=+3128.432703269" Jul 23 01:00:15.386293 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:15.386261 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78cf4c6f6-nqcdd_107a082e-c6f4-4d8d-a55a-763f1bedc830/volume-data-source-validator/0.log" Jul 23 01:00:16.433792 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:16.433750 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ndkl6_dd67adef-fce4-429b-b01b-8225f9c33ab4/dns/0.log" Jul 23 01:00:16.456900 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:16.456868 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ndkl6_dd67adef-fce4-429b-b01b-8225f9c33ab4/kube-rbac-proxy/0.log" Jul 23 01:00:16.598876 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:16.598846 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9stwt_7d457c50-6bca-4609-8897-cef340b087e8/dns-node-resolver/0.log" Jul 23 01:00:17.227273 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:17.227236 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5cc57669fb-7w5wx_1243e526-631f-487f-bcd1-a3a594979839/registry/0.log" Jul 23 01:00:17.275685 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:17.275654 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jctrc_aeaeb6df-4c50-464f-b5fa-a6e599d2fdf9/node-ca/0.log" Jul 23 01:00:18.289936 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:18.289900 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678bfjwm_a5b4d97f-58c7-4032-85f5-8c31e65c8947/istio-proxy/0.log" Jul 23 01:00:18.417637 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:18.417605 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-t62mx_557397d5-d277-4ee8-a5bd-bbc4fec09acb/discovery/0.log" Jul 23 01:00:18.592337 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:18.592242 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-glvxf_baedce3c-d14a-49ac-b183-8468e48c8d89/payload-processing/0.log" Jul 23 01:00:18.616395 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:18.616367 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-76974999cb-mggnm_c0ba4e0d-2b3e-4a20-8f3e-1296830be034/router/0.log" Jul 23 01:00:19.313238 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:19.313207 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pqp48_c1bab695-9ea2-4498-bf9e-e750b5850fcc/serve-healthcheck-canary/0.log" Jul 23 01:00:19.821350 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:19.821318 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-w24mg_ec2a61a7-0a82-4f9a-aac7-001dec5f5a6f/insights-operator/0.log" Jul 23 01:00:19.821950 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:19.821930 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-w24mg_ec2a61a7-0a82-4f9a-aac7-001dec5f5a6f/insights-operator/1.log" Jul 23 01:00:19.998892 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:19.998868 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nz9r9_462cefbf-6678-4aa7-96ba-66a0630f1a63/kube-rbac-proxy/0.log" Jul 23 01:00:20.022045 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:20.022014 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nz9r9_462cefbf-6678-4aa7-96ba-66a0630f1a63/exporter/0.log" Jul 23 01:00:20.045475 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:20.045444 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nz9r9_462cefbf-6678-4aa7-96ba-66a0630f1a63/extractor/0.log" Jul 23 01:00:20.634935 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:20.634903 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mtxqj/perf-node-gather-daemonset-sd5lh" Jul 23 01:00:22.950252 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:22.950171 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f65bfdcb7-k8wsd_48c0a1e1-c11f-405e-baeb-98d98897dd11/manager/0.log" Jul 23 01:00:22.968049 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:22.968015 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f65bfdcb7-k8wsd_48c0a1e1-c11f-405e-baeb-98d98897dd11/copy-manifests/0.log" Jul 23 01:00:23.538319 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:23.538285 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678bfjwm_a5b4d97f-58c7-4032-85f5-8c31e65c8947/istio-proxy/0.log" Jul 23 01:00:23.676275 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:23.676197 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-t62mx_557397d5-d277-4ee8-a5bd-bbc4fec09acb/discovery/0.log" Jul 23 01:00:23.806139 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:23.806100 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-glvxf_baedce3c-d14a-49ac-b183-8468e48c8d89/payload-processing/0.log" Jul 23 01:00:23.821347 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:23.821299 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-76974999cb-mggnm_c0ba4e0d-2b3e-4a20-8f3e-1296830be034/router/0.log" Jul 23 01:00:24.336417 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:24.336387 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-85d467fcb7-bn4gl_a17498fa-9c38-4f7c-958a-bb029dfdabac/manager/0.log" Jul 23 01:00:24.368368 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:24.368336 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-9qwh7_1dc803ca-ac1a-441e-8d4f-d146e45ac96a/openshift-lws-operator/0.log" Jul 23 01:00:25.009831 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:25.009801 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678bfjwm_a5b4d97f-58c7-4032-85f5-8c31e65c8947/istio-proxy/0.log" Jul 23 01:00:25.131123 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:25.131071 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-t62mx_557397d5-d277-4ee8-a5bd-bbc4fec09acb/discovery/0.log" Jul 23 01:00:25.270725 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:25.270698 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-glvxf_baedce3c-d14a-49ac-b183-8468e48c8d89/payload-processing/0.log" Jul 23 01:00:25.284558 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:25.284533 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-76974999cb-mggnm_c0ba4e0d-2b3e-4a20-8f3e-1296830be034/router/0.log" Jul 23 01:00:29.281197 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:29.281113 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-bs2ws_1b62b19c-4feb-48ef-807e-37741fa4db0b/migrator/0.log" Jul 23 01:00:29.303500 ip-10-0-131-121 kubenswrapper[2562]: I0723 01:00:29.303463 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-bs2ws_1b62b19c-4feb-48ef-807e-37741fa4db0b/graceful-termination/0.log"