Jul 28 22:09:38.581113 ip-10-0-142-132 systemd[1]: Starting Kubernetes Kubelet... Jul 28 22:09:38.968251 ip-10-0-142-132 kubenswrapper[2553]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 22:09:38.968251 ip-10-0-142-132 kubenswrapper[2553]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 28 22:09:38.968251 ip-10-0-142-132 kubenswrapper[2553]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 22:09:38.968251 ip-10-0-142-132 kubenswrapper[2553]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 28 22:09:38.968251 ip-10-0-142-132 kubenswrapper[2553]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 22:09:38.969264 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.969177 2553 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 28 22:09:38.973576 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973559 2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 22:09:38.973576 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973576 2553 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 22:09:38.973643 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973580 2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 22:09:38.973643 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973584 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 22:09:38.973643 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973587 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 22:09:38.973643 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973592 2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 22:09:38.973643 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973596 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 22:09:38.973643 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973599 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 22:09:38.973643 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973602 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 22:09:38.973643 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973605 2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 22:09:38.973643 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973608 2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 22:09:38.973643 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973612 2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 22:09:38.973643 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973615 2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 22:09:38.973643 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973618 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 22:09:38.973643 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973622 2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 22:09:38.973643 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973625 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 22:09:38.973643 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973629 2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 22:09:38.973643 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973633 2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 22:09:38.973643 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973636 2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 22:09:38.973643 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973639 2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 22:09:38.974092 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973641 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 22:09:38.974092 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973644 2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 22:09:38.974092 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973647 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 22:09:38.974092 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973650 2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 22:09:38.974092 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973653 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 22:09:38.974092 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973655 2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 22:09:38.974092 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973658 2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 22:09:38.974092 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973660 2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 22:09:38.974092 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973663 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 22:09:38.974092 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973666 2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 22:09:38.974092 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973668 2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 22:09:38.974092 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973671 2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 22:09:38.974092 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973673 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 22:09:38.974092 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973676 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 22:09:38.974092 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973678 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 22:09:38.974092 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973680 2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 22:09:38.974092 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973683 2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 22:09:38.974092 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973685 2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 22:09:38.974092 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973688 2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 22:09:38.974092 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973691 2553 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 22:09:38.974581 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973693 2553 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 22:09:38.974581 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973696 2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 22:09:38.974581 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973698 2553 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 22:09:38.974581 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973701 2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 22:09:38.974581 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973703 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 22:09:38.974581 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973706 2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 22:09:38.974581 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973715 2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 22:09:38.974581 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973718 2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 22:09:38.974581 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973720 2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 22:09:38.974581 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973722 2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 22:09:38.974581 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973725 2553 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 22:09:38.974581 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973728 2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 22:09:38.974581 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973730 2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 22:09:38.974581 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973735 2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 22:09:38.974581 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973738 2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 22:09:38.974581 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973740 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 22:09:38.974581 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973743 2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 22:09:38.974581 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973746 2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 22:09:38.974581 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973748 2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 22:09:38.974581 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973751 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 22:09:38.975091 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973753 2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 22:09:38.975091 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973756 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 22:09:38.975091 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973759 2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 22:09:38.975091 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973761 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 22:09:38.975091 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973764 2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 22:09:38.975091 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973766 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 22:09:38.975091 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973769 2553 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 22:09:38.975091 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973771 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 22:09:38.975091 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973774 2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 22:09:38.975091 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973776 2553 feature_gate.go:328] unrecognized feature gate: Example Jul 28 22:09:38.975091 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973779 2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 22:09:38.975091 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973782 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 22:09:38.975091 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973785 2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 22:09:38.975091 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973787 2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 22:09:38.975091 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973790 2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 22:09:38.975091 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973792 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 22:09:38.975091 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973795 2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 22:09:38.975091 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973797 2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 22:09:38.975091 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973799 2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 22:09:38.975091 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973802 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 22:09:38.975571 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973805 2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 22:09:38.975571 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973807 2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 22:09:38.975571 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973825 2553 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 22:09:38.975571 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973828 2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 22:09:38.975571 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973831 2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 22:09:38.975571 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973834 2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 22:09:38.975571 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973837 2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 22:09:38.975571 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973840 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 22:09:38.975571 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973842 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 22:09:38.975571 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973845 2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 22:09:38.975571 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973847 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 22:09:38.975571 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973850 2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 22:09:38.975571 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973852 2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 22:09:38.975571 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973855 2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 22:09:38.975571 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973858 2553 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 22:09:38.975571 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973860 2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 22:09:38.975571 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973863 2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 22:09:38.975571 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973865 2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 22:09:38.975571 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.973867 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 22:09:38.977183 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977166 2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 22:09:38.977183 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977181 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 22:09:38.977183 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977184 2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 22:09:38.977183 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977188 2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 22:09:38.977338 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977193 2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 22:09:38.977338 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977197 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 22:09:38.977338 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977200 2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 22:09:38.977338 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977203 2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 22:09:38.977338 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977206 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 22:09:38.977338 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977209 2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 22:09:38.977338 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977212 2553 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 22:09:38.977338 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977215 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 22:09:38.977338 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977218 2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 22:09:38.977338 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977221 2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 22:09:38.977338 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977223 2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 22:09:38.977338 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977226 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 22:09:38.977338 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977228 2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 22:09:38.977338 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977231 2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 22:09:38.977338 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977235 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 22:09:38.977338 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977237 2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 22:09:38.977338 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977240 2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 22:09:38.977338 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977252 2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 22:09:38.977338 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977255 2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 22:09:38.977838 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977258 2553 feature_gate.go:328] unrecognized feature gate: Example Jul 28 22:09:38.977838 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977260 2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 22:09:38.977838 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977263 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 22:09:38.977838 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977265 2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 22:09:38.977838 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977268 2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 22:09:38.977838 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977271 2553 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 22:09:38.977838 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977274 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 22:09:38.977838 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977276 2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 22:09:38.977838 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977279 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 22:09:38.977838 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977282 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 22:09:38.977838 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977284 2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 22:09:38.977838 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977286 2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 22:09:38.977838 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977289 2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 22:09:38.977838 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977292 2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 22:09:38.977838 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977295 2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 22:09:38.977838 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977298 2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 22:09:38.977838 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977300 2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 22:09:38.977838 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977303 2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 22:09:38.977838 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977305 2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 22:09:38.977838 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977308 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 22:09:38.978327 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977310 2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 22:09:38.978327 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977313 2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 22:09:38.978327 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977315 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 22:09:38.978327 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977317 2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 22:09:38.978327 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977320 2553 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 22:09:38.978327 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977322 2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 22:09:38.978327 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977324 2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 22:09:38.978327 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977327 2553 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 22:09:38.978327 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977329 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 22:09:38.978327 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977332 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 22:09:38.978327 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977334 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 22:09:38.978327 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977338 2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 22:09:38.978327 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977342 2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 22:09:38.978327 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977346 2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 22:09:38.978327 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977349 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 22:09:38.978327 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977352 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 22:09:38.978327 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977355 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 22:09:38.978327 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977359 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 22:09:38.978327 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977362 2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 22:09:38.978798 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977364 2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 22:09:38.978798 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977367 2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 22:09:38.978798 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977370 2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 22:09:38.978798 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977372 2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 22:09:38.978798 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977374 2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 22:09:38.978798 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977377 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 22:09:38.978798 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977380 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 22:09:38.978798 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977383 2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 22:09:38.978798 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977385 2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 22:09:38.978798 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977388 2553 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 22:09:38.978798 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977391 2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 22:09:38.978798 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977393 2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 22:09:38.978798 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977396 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 22:09:38.978798 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977398 2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 22:09:38.978798 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977400 2553 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 22:09:38.978798 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977403 2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 22:09:38.978798 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977405 2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 22:09:38.978798 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977408 2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 22:09:38.978798 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977410 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 22:09:38.978798 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977412 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 22:09:38.979312 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977415 2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 22:09:38.979312 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977418 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 22:09:38.979312 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977420 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 22:09:38.979312 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977423 2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 22:09:38.979312 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977425 2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 22:09:38.979312 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977428 2553 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 22:09:38.979312 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977431 2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 22:09:38.979312 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977433 2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 22:09:38.979312 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977436 2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 22:09:38.979312 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977438 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 22:09:38.979312 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977441 2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 22:09:38.979312 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977443 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 22:09:38.979312 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977446 2553 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 22:09:38.979312 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977448 2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 22:09:38.979312 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977451 2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 22:09:38.979312 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977453 2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 22:09:38.979312 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.977455 2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 22:09:38.979312 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978469 2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 22:09:38.979312 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978477 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 22:09:38.979312 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978481 2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 22:09:38.979802 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978484 2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 22:09:38.979802 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978487 2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 22:09:38.979802 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978490 2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 22:09:38.979802 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978493 2553 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 22:09:38.979802 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978496 2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 22:09:38.979802 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978498 2553 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 22:09:38.979802 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978501 2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 22:09:38.979802 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978504 2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 22:09:38.979802 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978506 2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 22:09:38.979802 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978509 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 22:09:38.979802 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978511 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 22:09:38.979802 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978514 2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 22:09:38.979802 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978516 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 22:09:38.979802 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978519 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 22:09:38.979802 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978522 2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 22:09:38.979802 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978524 2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 22:09:38.979802 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978527 2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 22:09:38.979802 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978529 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 22:09:38.979802 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978532 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 22:09:38.979802 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978534 2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 22:09:38.980305 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978537 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 22:09:38.980305 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978540 2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 22:09:38.980305 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978542 2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 22:09:38.980305 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978545 2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 22:09:38.980305 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978547 2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 22:09:38.980305 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978550 2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 22:09:38.980305 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978552 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 22:09:38.980305 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978555 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 22:09:38.980305 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978557 2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 22:09:38.980305 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978559 2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 22:09:38.980305 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978562 2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 22:09:38.980305 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978565 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 22:09:38.980305 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978567 2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 22:09:38.980305 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978570 2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 22:09:38.980305 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978572 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 22:09:38.980305 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978576 2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 22:09:38.980305 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978580 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 22:09:38.980305 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978583 2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 22:09:38.980305 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978586 2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 22:09:38.980762 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978588 2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 22:09:38.980762 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978591 2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 22:09:38.980762 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978593 2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 22:09:38.980762 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978596 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 22:09:38.980762 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978599 2553 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 22:09:38.980762 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978601 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 22:09:38.980762 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978603 2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 22:09:38.980762 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978607 2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 22:09:38.980762 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978609 2553 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 22:09:38.980762 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978612 2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 22:09:38.980762 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978614 2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 22:09:38.980762 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978617 2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 22:09:38.980762 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978619 2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 22:09:38.980762 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978622 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 22:09:38.980762 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978624 2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 22:09:38.980762 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978627 2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 22:09:38.980762 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978630 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 22:09:38.980762 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978632 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 22:09:38.980762 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978634 2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 22:09:38.981243 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978637 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 22:09:38.981243 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978639 2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 22:09:38.981243 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978641 2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 22:09:38.981243 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978644 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 22:09:38.981243 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978646 2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 22:09:38.981243 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978649 2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 22:09:38.981243 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978651 2553 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 22:09:38.981243 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978654 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 22:09:38.981243 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978656 2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 22:09:38.981243 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978659 2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 22:09:38.981243 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978661 2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 22:09:38.981243 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978663 2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 22:09:38.981243 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978666 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 22:09:38.981243 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978668 2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 22:09:38.981243 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978671 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 22:09:38.981243 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978673 2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 22:09:38.981243 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978676 2553 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 22:09:38.981243 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978678 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 22:09:38.981243 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978680 2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 22:09:38.981243 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978684 2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 22:09:38.981763 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978686 2553 feature_gate.go:328] unrecognized feature gate: Example Jul 28 22:09:38.981763 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978689 2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 22:09:38.981763 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978692 2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 22:09:38.981763 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978694 2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 22:09:38.981763 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978696 2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 22:09:38.981763 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978699 2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 22:09:38.981763 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978701 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 22:09:38.981763 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978704 2553 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 22:09:38.981763 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978708 2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 22:09:38.981763 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978711 2553 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 22:09:38.981763 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978714 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 22:09:38.981763 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978717 2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 22:09:38.981763 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978720 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 22:09:38.981763 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978723 2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 22:09:38.981763 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978727 2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 22:09:38.981763 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978730 2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 22:09:38.981763 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978732 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 22:09:38.981763 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.978735 2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 22:09:38.981763 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.978804 2553 flags.go:64] FLAG: --address="0.0.0.0" Jul 28 22:09:38.981763 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.978831 2553 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 28 22:09:38.982249 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.978840 2553 flags.go:64] FLAG: --anonymous-auth="true" Jul 28 22:09:38.982249 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.978845 2553 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 28 22:09:38.982249 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.978849 2553 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 28 22:09:38.982249 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.978852 2553 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 28 22:09:38.982249 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.978861 2553 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 28 22:09:38.982249 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.978865 2553 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 28 22:09:38.982249 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.978868 2553 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 28 22:09:38.982249 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.978871 2553 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 28 22:09:38.982249 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.978874 2553 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 28 22:09:38.982249 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.978877 2553 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 28 22:09:38.982249 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.978880 2553 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 28 22:09:38.982249 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.978885 2553 flags.go:64] FLAG: --cgroup-root="" Jul 28 22:09:38.982249 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.978888 2553 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 28 22:09:38.982249 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.978891 2553 flags.go:64] FLAG: --client-ca-file="" Jul 28 22:09:38.982249 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.978895 2553 flags.go:64] FLAG: --cloud-provider="external" Jul 28 22:09:38.982249 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.978898 2553 flags.go:64] FLAG: --cluster-dns="[]" Jul 28 22:09:38.982249 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.978902 2553 flags.go:64] FLAG: --cluster-domain="" Jul 28 22:09:38.982249 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.978904 2553 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 28 22:09:38.982249 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.978907 2553 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 28 22:09:38.982249 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.978910 2553 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 28 22:09:38.982249 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.978913 2553 flags.go:64] FLAG: --container-log-max-files="5" Jul 28 22:09:38.982249 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.978917 2553 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 28 22:09:38.982249 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.978920 2553 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 28 22:09:38.982856 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.978922 2553 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 28 22:09:38.982856 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.978926 2553 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 28 22:09:38.982856 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.978928 2553 flags.go:64] FLAG: --contention-profiling="false" Jul 28 22:09:38.982856 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.978931 2553 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 28 22:09:38.982856 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.978934 2553 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 28 22:09:38.982856 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.978937 2553 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 28 22:09:38.982856 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.978940 2553 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 28 22:09:38.982856 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.978944 2553 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 28 22:09:38.982856 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.978947 2553 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 28 22:09:38.982856 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.978950 2553 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 28 22:09:38.982856 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.978952 2553 flags.go:64] FLAG: --enable-load-reader="false" Jul 28 22:09:38.982856 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.978955 2553 flags.go:64] FLAG: --enable-server="true" Jul 28 22:09:38.982856 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.978958 2553 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 28 22:09:38.982856 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.978962 2553 flags.go:64] FLAG: --event-burst="100" Jul 28 22:09:38.982856 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.978965 2553 flags.go:64] FLAG: --event-qps="50" Jul 28 22:09:38.982856 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.978968 2553 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 28 22:09:38.982856 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.978971 2553 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 28 22:09:38.982856 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.978974 2553 flags.go:64] FLAG: --eviction-hard="" Jul 28 22:09:38.982856 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.978977 2553 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 28 22:09:38.982856 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.978980 2553 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 28 22:09:38.982856 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.978984 2553 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 28 22:09:38.982856 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.978987 2553 flags.go:64] FLAG: --eviction-soft="" Jul 28 22:09:38.982856 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.978990 2553 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 28 22:09:38.982856 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.978994 2553 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 28 22:09:38.982856 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.978998 2553 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 28 22:09:38.983484 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979001 2553 flags.go:64] FLAG: --experimental-mounter-path="" Jul 28 22:09:38.983484 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979004 2553 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 28 22:09:38.983484 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979009 2553 flags.go:64] FLAG: --fail-swap-on="true" Jul 28 22:09:38.983484 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979012 2553 flags.go:64] FLAG: --feature-gates="" Jul 28 22:09:38.983484 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979016 2553 flags.go:64] FLAG: --file-check-frequency="20s" Jul 28 22:09:38.983484 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979018 2553 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 28 22:09:38.983484 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979021 2553 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 28 22:09:38.983484 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979025 2553 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 28 22:09:38.983484 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979028 2553 flags.go:64] FLAG: --healthz-port="10248" Jul 28 22:09:38.983484 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979031 2553 flags.go:64] FLAG: --help="false" Jul 28 22:09:38.983484 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979033 2553 flags.go:64] FLAG: --hostname-override="ip-10-0-142-132.ec2.internal" Jul 28 22:09:38.983484 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979036 2553 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 28 22:09:38.983484 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979039 2553 flags.go:64] FLAG: --http-check-frequency="20s" Jul 28 22:09:38.983484 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979042 2553 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 28 22:09:38.983484 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979045 2553 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 28 22:09:38.983484 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979048 2553 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 28 22:09:38.983484 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979051 2553 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 28 22:09:38.983484 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979054 2553 flags.go:64] FLAG: --image-service-endpoint="" Jul 28 22:09:38.983484 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979057 2553 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 28 22:09:38.983484 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979060 2553 flags.go:64] FLAG: --kube-api-burst="100" Jul 28 22:09:38.983484 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979062 2553 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 28 22:09:38.983484 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979065 2553 flags.go:64] FLAG: --kube-api-qps="50" Jul 28 22:09:38.983484 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979068 2553 flags.go:64] FLAG: --kube-reserved="" Jul 28 22:09:38.983484 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979071 2553 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 28 22:09:38.984089 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979074 2553 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 28 22:09:38.984089 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979076 2553 flags.go:64] FLAG: --kubelet-cgroups="" Jul 28 22:09:38.984089 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979079 2553 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 28 22:09:38.984089 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979083 2553 flags.go:64] FLAG: --lock-file="" Jul 28 22:09:38.984089 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979086 2553 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 28 22:09:38.984089 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979088 2553 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 28 22:09:38.984089 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979091 2553 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 28 22:09:38.984089 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979099 2553 flags.go:64] FLAG: --log-json-split-stream="false" Jul 28 22:09:38.984089 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979102 2553 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 28 22:09:38.984089 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979105 2553 flags.go:64] FLAG: --log-text-split-stream="false" Jul 28 22:09:38.984089 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979109 2553 flags.go:64] FLAG: --logging-format="text" Jul 28 22:09:38.984089 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979112 2553 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 28 22:09:38.984089 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979115 2553 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 28 22:09:38.984089 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979118 2553 flags.go:64] FLAG: --manifest-url="" Jul 28 22:09:38.984089 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979121 2553 flags.go:64] FLAG: --manifest-url-header="" Jul 28 22:09:38.984089 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979125 2553 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 28 22:09:38.984089 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979128 2553 flags.go:64] FLAG: --max-open-files="1000000" Jul 28 22:09:38.984089 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979132 2553 flags.go:64] FLAG: --max-pods="110" Jul 28 22:09:38.984089 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979134 2553 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 28 22:09:38.984089 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979137 2553 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 28 22:09:38.984089 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979140 2553 flags.go:64] FLAG: --memory-manager-policy="None" Jul 28 22:09:38.984089 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979143 2553 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 28 22:09:38.984089 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979145 2553 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 28 22:09:38.984089 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979148 2553 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 28 22:09:38.984089 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979151 2553 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 28 22:09:38.984704 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979159 2553 flags.go:64] FLAG: --node-status-max-images="50" Jul 28 22:09:38.984704 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979161 2553 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 28 22:09:38.984704 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979164 2553 flags.go:64] FLAG: --oom-score-adj="-999" Jul 28 22:09:38.984704 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979167 2553 flags.go:64] FLAG: --pod-cidr="" Jul 28 22:09:38.984704 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979170 2553 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 28 22:09:38.984704 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979176 2553 flags.go:64] FLAG: --pod-manifest-path="" Jul 28 22:09:38.984704 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979179 2553 flags.go:64] FLAG: --pod-max-pids="-1" Jul 28 22:09:38.984704 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979182 2553 flags.go:64] FLAG: --pods-per-core="0" Jul 28 22:09:38.984704 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979185 2553 flags.go:64] FLAG: --port="10250" Jul 28 22:09:38.984704 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979187 2553 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 28 22:09:38.984704 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979192 2553 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-049ccc4f4e34a107f" Jul 28 22:09:38.984704 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979195 2553 flags.go:64] FLAG: --qos-reserved="" Jul 28 22:09:38.984704 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979198 2553 flags.go:64] FLAG: --read-only-port="10255" Jul 28 22:09:38.984704 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979201 2553 flags.go:64] FLAG: --register-node="true" Jul 28 22:09:38.984704 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979204 2553 flags.go:64] FLAG: --register-with-taints="" Jul 28 22:09:38.984704 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979207 2553 flags.go:64] FLAG: --registry-burst="10" Jul 28 22:09:38.984704 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979211 2553 flags.go:64] FLAG: --registry-qps="5" Jul 28 22:09:38.984704 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979214 2553 flags.go:64] FLAG: --reserved-cpus="" Jul 28 22:09:38.984704 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979218 2553 flags.go:64] FLAG: --reserved-memory="" Jul 28 22:09:38.984704 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979221 2553 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 28 22:09:38.984704 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979224 2553 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 28 22:09:38.984704 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979227 2553 flags.go:64] FLAG: --rotate-certificates="false" Jul 28 22:09:38.984704 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979230 2553 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 28 22:09:38.984704 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979233 2553 flags.go:64] FLAG: --runonce="false" Jul 28 22:09:38.984704 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979235 2553 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 28 22:09:38.985366 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979238 2553 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 28 22:09:38.985366 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979241 2553 flags.go:64] FLAG: --seccomp-default="false" Jul 28 22:09:38.985366 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979244 2553 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 28 22:09:38.985366 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979247 2553 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 28 22:09:38.985366 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979250 2553 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 28 22:09:38.985366 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979253 2553 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 28 22:09:38.985366 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979256 2553 flags.go:64] FLAG: --storage-driver-password="root" Jul 28 22:09:38.985366 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979258 2553 flags.go:64] FLAG: --storage-driver-secure="false" Jul 28 22:09:38.985366 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979261 2553 flags.go:64] FLAG: --storage-driver-table="stats" Jul 28 22:09:38.985366 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979264 2553 flags.go:64] FLAG: --storage-driver-user="root" Jul 28 22:09:38.985366 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979266 2553 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 28 22:09:38.985366 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979270 2553 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 28 22:09:38.985366 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979272 2553 flags.go:64] FLAG: --system-cgroups="" Jul 28 22:09:38.985366 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979275 2553 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 28 22:09:38.985366 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979280 2553 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 28 22:09:38.985366 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979283 2553 flags.go:64] FLAG: --tls-cert-file="" Jul 28 22:09:38.985366 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979286 2553 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 28 22:09:38.985366 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979291 2553 flags.go:64] FLAG: --tls-min-version="" Jul 28 22:09:38.985366 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979293 2553 flags.go:64] FLAG: --tls-private-key-file="" Jul 28 22:09:38.985366 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979296 2553 flags.go:64] FLAG: --topology-manager-policy="none" Jul 28 22:09:38.985366 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979298 2553 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 28 22:09:38.985366 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979301 2553 flags.go:64] FLAG: --topology-manager-scope="container" Jul 28 22:09:38.985366 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979304 2553 flags.go:64] FLAG: --v="2" Jul 28 22:09:38.985366 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979308 2553 flags.go:64] FLAG: --version="false" Jul 28 22:09:38.985366 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979312 2553 flags.go:64] FLAG: --vmodule="" Jul 28 22:09:38.986011 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979317 2553 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 28 22:09:38.986011 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.979320 2553 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 28 22:09:38.986011 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979451 2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 22:09:38.986011 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979455 2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 22:09:38.986011 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979458 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 22:09:38.986011 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979461 2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 22:09:38.986011 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979464 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 22:09:38.986011 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979467 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 22:09:38.986011 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979469 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 22:09:38.986011 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979472 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 22:09:38.986011 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979474 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 22:09:38.986011 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979477 2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 22:09:38.986011 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979480 2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 22:09:38.986011 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979483 2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 22:09:38.986011 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979485 2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 22:09:38.986011 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979487 2553 feature_gate.go:328] unrecognized feature gate: Example Jul 28 22:09:38.986011 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979490 2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 22:09:38.986011 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979492 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 22:09:38.986011 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979494 2553 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 22:09:38.986011 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979497 2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 22:09:38.986510 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979499 2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 22:09:38.986510 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979502 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 22:09:38.986510 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979504 2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 22:09:38.986510 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979507 2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 22:09:38.986510 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979511 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 22:09:38.986510 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979514 2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 22:09:38.986510 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979518 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 22:09:38.986510 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979521 2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 22:09:38.986510 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979523 2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 22:09:38.986510 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979526 2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 22:09:38.986510 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979529 2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 22:09:38.986510 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979531 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 22:09:38.986510 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979536 2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 22:09:38.986510 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979538 2553 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 22:09:38.986510 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979541 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 22:09:38.986510 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979544 2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 22:09:38.986510 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979546 2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 22:09:38.986510 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979549 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 22:09:38.986510 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979551 2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 22:09:38.987000 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979554 2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 22:09:38.987000 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979556 2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 22:09:38.987000 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979559 2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 22:09:38.987000 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979561 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 22:09:38.987000 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979564 2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 22:09:38.987000 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979566 2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 22:09:38.987000 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979569 2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 22:09:38.987000 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979572 2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 22:09:38.987000 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979574 2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 22:09:38.987000 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979576 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 22:09:38.987000 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979579 2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 22:09:38.987000 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979581 2553 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 22:09:38.987000 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979584 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 22:09:38.987000 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979586 2553 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 22:09:38.987000 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979589 2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 22:09:38.987000 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979591 2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 22:09:38.987000 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979593 2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 22:09:38.987000 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979598 2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 22:09:38.987000 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979600 2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 22:09:38.987465 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979602 2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 22:09:38.987465 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979605 2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 22:09:38.987465 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979607 2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 22:09:38.987465 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979610 2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 22:09:38.987465 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979612 2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 22:09:38.987465 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979615 2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 22:09:38.987465 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979620 2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 22:09:38.987465 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979624 2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 22:09:38.987465 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979627 2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 22:09:38.987465 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979629 2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 22:09:38.987465 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979632 2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 22:09:38.987465 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979635 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 22:09:38.987465 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979638 2553 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 22:09:38.987465 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979641 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 22:09:38.987465 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979644 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 22:09:38.987465 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979646 2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 22:09:38.987465 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979649 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 22:09:38.987465 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979651 2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 22:09:38.987465 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979653 2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 22:09:38.987465 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979656 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 22:09:38.987973 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979658 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 22:09:38.987973 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979661 2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 22:09:38.987973 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979663 2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 22:09:38.987973 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979665 2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 22:09:38.987973 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979668 2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 22:09:38.987973 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979670 2553 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 22:09:38.987973 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979674 2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 22:09:38.987973 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979677 2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 22:09:38.987973 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979680 2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 22:09:38.987973 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979683 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 22:09:38.987973 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979687 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 22:09:38.987973 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979690 2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 22:09:38.987973 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979692 2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 22:09:38.987973 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979695 2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 22:09:38.987973 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979698 2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 22:09:38.987973 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979700 2553 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 22:09:38.987973 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979703 2553 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 22:09:38.987973 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979705 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 22:09:38.987973 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979709 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 22:09:38.987973 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979712 2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 22:09:38.988456 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979715 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 22:09:38.988456 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979718 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 22:09:38.988456 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.979721 2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 22:09:38.988456 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.980323 2553 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 22:09:38.988456 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.987478 2553 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 28 22:09:38.988456 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.987494 2553 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 28 22:09:38.988456 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.988127 2553 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 28 22:09:38.988456 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.988135 2553 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 28 22:09:38.988456 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988192 2553 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 22:09:38.988456 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988197 2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 22:09:38.988456 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988201 2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 22:09:38.988456 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988204 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 22:09:38.988456 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988207 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 22:09:38.988456 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988210 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 22:09:38.988456 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988213 2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 22:09:38.988899 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988215 2553 feature_gate.go:328] unrecognized feature gate: Example Jul 28 22:09:38.988899 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988218 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 22:09:38.988899 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988221 2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 22:09:38.988899 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988223 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 22:09:38.988899 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988226 2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 22:09:38.988899 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988228 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 22:09:38.988899 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988230 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 22:09:38.988899 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988233 2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 22:09:38.988899 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988236 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 22:09:38.988899 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988238 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 22:09:38.988899 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988240 2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 22:09:38.988899 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988243 2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 22:09:38.988899 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988246 2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 22:09:38.988899 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988248 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 22:09:38.988899 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988251 2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 22:09:38.988899 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988253 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 22:09:38.988899 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988256 2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 22:09:38.988899 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988258 2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 22:09:38.988899 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988260 2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 22:09:38.989357 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988263 2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 22:09:38.989357 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988266 2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 22:09:38.989357 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988268 2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 22:09:38.989357 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988272 2553 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 22:09:38.989357 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988274 2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 22:09:38.989357 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988277 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 22:09:38.989357 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988280 2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 22:09:38.989357 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988282 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 22:09:38.989357 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988284 2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 22:09:38.989357 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988287 2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 22:09:38.989357 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988290 2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 22:09:38.989357 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988292 2553 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 22:09:38.989357 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988295 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 22:09:38.989357 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988297 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 22:09:38.989357 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988300 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 22:09:38.989357 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988302 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 22:09:38.989357 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988305 2553 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 22:09:38.989357 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988308 2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 22:09:38.989357 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988310 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 22:09:38.989357 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988313 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 22:09:38.989940 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988315 2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 22:09:38.989940 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988318 2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 22:09:38.989940 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988320 2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 22:09:38.989940 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988323 2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 22:09:38.989940 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988325 2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 22:09:38.989940 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988327 2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 22:09:38.989940 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988332 2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 22:09:38.989940 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988335 2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 22:09:38.989940 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988338 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 22:09:38.989940 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988341 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 22:09:38.989940 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988344 2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 22:09:38.989940 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988346 2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 22:09:38.989940 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988350 2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 22:09:38.989940 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988352 2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 22:09:38.989940 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988355 2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 22:09:38.989940 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988359 2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 22:09:38.989940 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988366 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 22:09:38.989940 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988369 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 22:09:38.989940 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988372 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 22:09:38.990421 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988375 2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 22:09:38.990421 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988379 2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 22:09:38.990421 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988383 2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 22:09:38.990421 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988386 2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 22:09:38.990421 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988388 2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 22:09:38.990421 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988391 2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 22:09:38.990421 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988393 2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 22:09:38.990421 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988396 2553 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 22:09:38.990421 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988398 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 22:09:38.990421 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988402 2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 22:09:38.990421 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988405 2553 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 22:09:38.990421 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988408 2553 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 22:09:38.990421 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988411 2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 22:09:38.990421 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988413 2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 22:09:38.990421 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988416 2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 22:09:38.990421 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988419 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 22:09:38.990421 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988422 2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 22:09:38.990421 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988424 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 22:09:38.990421 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988427 2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 22:09:38.990914 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988430 2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 22:09:38.990914 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988432 2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 22:09:38.990914 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988435 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 22:09:38.990914 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988437 2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 22:09:38.990914 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988439 2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 22:09:38.990914 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988442 2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 22:09:38.990914 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988444 2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 22:09:38.990914 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988447 2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 22:09:38.990914 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988449 2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 22:09:38.990914 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988453 2553 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 22:09:38.990914 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988455 2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 22:09:38.990914 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988459 2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 22:09:38.990914 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988461 2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 22:09:38.990914 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988464 2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 22:09:38.990914 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988467 2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 22:09:38.991299 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.988472 2553 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 22:09:38.991299 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988581 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 22:09:38.991299 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988586 2553 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 22:09:38.991299 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988589 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 22:09:38.991299 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988591 2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 22:09:38.991299 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988594 2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 22:09:38.991299 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988597 2553 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 22:09:38.991299 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988600 2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 22:09:38.991299 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988602 2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 22:09:38.991299 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988605 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 22:09:38.991299 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988607 2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 22:09:38.991299 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988610 2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 22:09:38.991299 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988612 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 22:09:38.991299 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988615 2553 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 22:09:38.991299 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988617 2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 22:09:38.991665 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988620 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 22:09:38.991665 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988622 2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 22:09:38.991665 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988624 2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 22:09:38.991665 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988627 2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 22:09:38.991665 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988630 2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 22:09:38.991665 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988632 2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 22:09:38.991665 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988634 2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 22:09:38.991665 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988637 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 22:09:38.991665 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988639 2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 22:09:38.991665 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988642 2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 22:09:38.991665 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988645 2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 22:09:38.991665 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988650 2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 22:09:38.991665 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988653 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 22:09:38.991665 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988656 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 22:09:38.991665 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988662 2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 22:09:38.991665 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988665 2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 22:09:38.991665 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988668 2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 22:09:38.991665 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988671 2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 22:09:38.992115 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988673 2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 22:09:38.992115 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988675 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 22:09:38.992115 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988678 2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 22:09:38.992115 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988681 2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 22:09:38.992115 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988685 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 22:09:38.992115 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988688 2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 22:09:38.992115 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988690 2553 feature_gate.go:328] unrecognized feature gate: Example Jul 28 22:09:38.992115 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988693 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 22:09:38.992115 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988695 2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 22:09:38.992115 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988697 2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 22:09:38.992115 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988700 2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 22:09:38.992115 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988702 2553 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 22:09:38.992115 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988704 2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 22:09:38.992115 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988707 2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 22:09:38.992115 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988709 2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 22:09:38.992115 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988712 2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 22:09:38.992115 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988714 2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 22:09:38.992115 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988717 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 22:09:38.992115 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988719 2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 22:09:38.992115 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988722 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 22:09:38.992595 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988724 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 22:09:38.992595 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988727 2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 22:09:38.992595 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988729 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 22:09:38.992595 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988732 2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 22:09:38.992595 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988734 2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 22:09:38.992595 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988738 2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 22:09:38.992595 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988740 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 22:09:38.992595 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988742 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 22:09:38.992595 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988745 2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 22:09:38.992595 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988752 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 22:09:38.992595 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988755 2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 22:09:38.992595 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988757 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 22:09:38.992595 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988760 2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 22:09:38.992595 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988762 2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 22:09:38.992595 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988765 2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 22:09:38.992595 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988767 2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 22:09:38.992595 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988770 2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 22:09:38.992595 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988772 2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 22:09:38.992595 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988774 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 22:09:38.993073 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988777 2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 22:09:38.993073 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988779 2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 22:09:38.993073 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988781 2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 22:09:38.993073 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988784 2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 22:09:38.993073 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988786 2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 22:09:38.993073 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988789 2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 22:09:38.993073 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988791 2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 22:09:38.993073 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988794 2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 22:09:38.993073 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988796 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 22:09:38.993073 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988799 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 22:09:38.993073 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988801 2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 22:09:38.993073 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988804 2553 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 22:09:38.993073 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988806 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 22:09:38.993073 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988823 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 22:09:38.993073 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988827 2553 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 22:09:38.993073 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988831 2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 22:09:38.993073 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988834 2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 22:09:38.993073 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988836 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 22:09:38.993073 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988839 2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 22:09:38.993073 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988841 2553 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 22:09:38.993549 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988844 2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 22:09:38.993549 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988846 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 22:09:38.993549 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988848 2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 22:09:38.993549 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988852 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 22:09:38.993549 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988855 2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 22:09:38.993549 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988857 2553 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 22:09:38.993549 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988859 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 22:09:38.993549 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:38.988862 2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 22:09:38.993549 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.988867 2553 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 22:09:38.993549 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.988985 2553 server.go:962] "Client rotation is on, will bootstrap in background" Jul 28 22:09:38.993549 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.990863 2553 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 28 22:09:38.993549 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.992293 2553 server.go:1019] "Starting client certificate rotation" Jul 28 22:09:38.993549 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.992387 2553 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 28 22:09:38.993549 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.992419 2553 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 28 22:09:38.994463 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.994450 2553 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 28 22:09:38.995557 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:38.995538 2553 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 28 22:09:39.007408 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.007382 2553 log.go:25] "Validated CRI v1 runtime API" Jul 28 22:09:39.010902 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.010886 2553 log.go:25] "Validated CRI v1 image API" Jul 28 22:09:39.011733 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.011709 2553 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 28 22:09:39.014140 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.014118 2553 fs.go:135] Filesystem UUIDs: map[1b8a4b6f-8878-41a8-ba7c-52aaff712fb1:/dev/nvme0n1p3 5d2d6cda-ff32-4492-b703-9326e9391ea2:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Jul 28 22:09:39.014201 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.014139 2553 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 28 22:09:39.019761 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.019653 2553 manager.go:216] Machine: {Timestamp:2026-07-28 22:09:39.017854966 +0000 UTC m=+0.314003815 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098385 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2baf31588580ff06a9e48eae63375d SystemUUID:ec2baf31-5885-80ff-06a9-e48eae63375d BootID:56eb40cd-3ad8-4003-9b11-968c5b9acdeb Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:e9:a5:12:f2:5b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:e9:a5:12:f2:5b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:de:c1:34:a1:c8:b3 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 28 22:09:39.019761 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.019756 2553 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 28 22:09:39.019886 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.019852 2553 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 28 22:09:39.022342 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.022314 2553 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 28 22:09:39.022490 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.022345 2553 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-132.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 28 22:09:39.023022 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.023011 2553 topology_manager.go:138] "Creating topology manager with none policy" Jul 28 22:09:39.023059 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.023025 2553 container_manager_linux.go:309] "Creating device plugin manager" Jul 28 22:09:39.023059 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.023032 2553 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 28 22:09:39.023754 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.023744 2553 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 28 22:09:39.024421 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.024410 2553 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 28 22:09:39.024694 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.024678 2553 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 28 22:09:39.025994 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.025983 2553 state_mem.go:36] "Initialized new in-memory state store" Jul 28 22:09:39.026120 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.026110 2553 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 28 22:09:39.026149 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.026141 2553 kubelet.go:486] "Attempting to sync node with API server" Jul 28 22:09:39.026186 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.026150 2553 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 28 22:09:39.026186 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.026165 2553 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 28 22:09:39.026186 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.026178 2553 kubelet.go:398] "Adding apiserver pod source" Jul 28 22:09:39.026283 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.026188 2553 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 28 22:09:39.026635 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.026623 2553 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 28 22:09:39.026676 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.026644 2553 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 28 22:09:39.028766 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.028746 2553 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 28 22:09:39.030483 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.030467 2553 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 28 22:09:39.030545 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.030501 2553 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 28 22:09:39.032348 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.032330 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 28 22:09:39.032408 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.032356 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 28 22:09:39.032408 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.032366 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 28 22:09:39.032408 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.032374 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 28 22:09:39.032408 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.032387 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 28 22:09:39.032408 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.032397 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 28 22:09:39.032408 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.032408 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 28 22:09:39.032574 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.032416 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 28 22:09:39.032574 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.032427 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 28 22:09:39.032574 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.032435 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 28 22:09:39.032574 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.032447 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 28 22:09:39.032574 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.032457 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 28 22:09:39.033305 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.033293 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 28 22:09:39.033352 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.033307 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 28 22:09:39.036689 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.036675 2553 server.go:1268] "Started kubelet" Jul 28 22:09:39.036762 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.036731 2553 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 28 22:09:39.037289 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.037021 2553 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 28 22:09:39.037608 ip-10-0-142-132 systemd[1]: Started Kubernetes Kubelet. Jul 28 22:09:39.038677 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.038640 2553 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 28 22:09:39.039575 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:39.039544 2553 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 28 22:09:39.039693 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:39.039665 2553 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-132.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 28 22:09:39.040080 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.039779 2553 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-132.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 28 22:09:39.040664 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.040373 2553 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 28 22:09:39.041412 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.041386 2553 server.go:310] "Adding debug handlers to kubelet server" Jul 28 22:09:39.042973 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:39.042944 2553 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 28 22:09:39.045926 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:39.045005 2553 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-132.ec2.internal.18c69396ff4dc143 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-132.ec2.internal,UID:ip-10-0-142-132.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-132.ec2.internal,},FirstTimestamp:2026-07-28 22:09:39.036643651 +0000 UTC m=+0.332792501,LastTimestamp:2026-07-28 22:09:39.036643651 +0000 UTC m=+0.332792501,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-132.ec2.internal,}" Jul 28 22:09:39.048116 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.048100 2553 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 28 22:09:39.048190 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.048111 2553 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 28 22:09:39.048190 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:39.048184 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-132.ec2.internal\" not found" Jul 28 22:09:39.048613 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.048585 2553 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 28 22:09:39.048694 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.048627 2553 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 28 22:09:39.048694 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.048584 2553 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 28 22:09:39.048793 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.048708 2553 reconstruct.go:97] "Volume reconstruction finished" Jul 28 22:09:39.048793 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.048716 2553 reconciler.go:29] "Reconciler: start to sync state" Jul 28 22:09:39.049299 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.049285 2553 factory.go:153] Registering CRI-O factory Jul 28 22:09:39.049386 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.049318 2553 factory.go:223] Registration of the crio container factory successfully Jul 28 22:09:39.049386 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.049363 2553 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 28 22:09:39.049386 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.049370 2553 factory.go:55] Registering systemd factory Jul 28 22:09:39.049386 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.049375 2553 factory.go:223] Registration of the systemd container factory successfully Jul 28 22:09:39.049518 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.049394 2553 factory.go:103] Registering Raw factory Jul 28 22:09:39.049518 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.049408 2553 manager.go:1196] Started watching for new ooms in manager Jul 28 22:09:39.049783 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.049771 2553 manager.go:349] Starting recovery of all containers Jul 28 22:09:39.058948 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:39.058772 2553 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-132.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 28 22:09:39.059048 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:39.058949 2553 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 28 22:09:39.061418 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.061402 2553 manager.go:354] Recovery completed Jul 28 22:09:39.062679 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:39.062663 2553 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jul 28 22:09:39.065751 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.065737 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 22:09:39.068067 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.068047 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-132.ec2.internal" event="NodeHasSufficientMemory" Jul 28 22:09:39.068144 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.068082 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-132.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 22:09:39.068144 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.068095 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-132.ec2.internal" event="NodeHasSufficientPID" Jul 28 22:09:39.068649 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.068634 2553 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 28 22:09:39.068649 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.068646 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 22:09:39.068755 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.068648 2553 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 28 22:09:39.068755 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.068687 2553 state_mem.go:36] "Initialized new in-memory state store" Jul 28 22:09:39.069324 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.069308 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-132.ec2.internal" event="NodeHasSufficientMemory" Jul 28 22:09:39.069386 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.069337 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-132.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 22:09:39.069386 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.069349 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-132.ec2.internal" event="NodeHasSufficientPID" Jul 28 22:09:39.070080 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.070068 2553 policy_none.go:49] "None policy: Start" Jul 28 22:09:39.070122 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.070084 2553 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 28 22:09:39.070122 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.070094 2553 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 28 22:09:39.070876 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.070866 2553 policy_none.go:47] "Start" Jul 28 22:09:39.075517 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.075501 2553 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sxqvf" Jul 28 22:09:39.077376 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:39.077297 2553 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-132.ec2.internal.18c69397012d3ba5 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-132.ec2.internal,UID:ip-10-0-142-132.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-132.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-132.ec2.internal,},FirstTimestamp:2026-07-28 22:09:39.068066725 +0000 UTC m=+0.364215575,LastTimestamp:2026-07-28 22:09:39.068066725 +0000 UTC m=+0.364215575,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-132.ec2.internal,}" Jul 28 22:09:39.085010 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.084992 2553 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sxqvf" Jul 28 22:09:39.086892 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:39.086806 2553 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-132.ec2.internal.18c69397012d935e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-132.ec2.internal,UID:ip-10-0-142-132.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-142-132.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-142-132.ec2.internal,},FirstTimestamp:2026-07-28 22:09:39.068089182 +0000 UTC m=+0.364238033,LastTimestamp:2026-07-28 22:09:39.068089182 +0000 UTC m=+0.364238033,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-132.ec2.internal,}" Jul 28 22:09:39.116398 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.116377 2553 manager.go:337] "Starting Device Plugin manager" Jul 28 22:09:39.116495 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:39.116416 2553 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 28 22:09:39.116495 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.116426 2553 server.go:87] "Starting device plugin registration server" Jul 28 22:09:39.116674 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.116661 2553 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 28 22:09:39.116719 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.116676 2553 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 28 22:09:39.116916 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.116888 2553 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 28 22:09:39.116988 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.116975 2553 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 28 22:09:39.116988 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.116985 2553 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 28 22:09:39.117387 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:39.117369 2553 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 28 22:09:39.117464 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:39.117432 2553 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-132.ec2.internal\" not found" Jul 28 22:09:39.188157 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.188114 2553 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 28 22:09:39.189482 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.189464 2553 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 28 22:09:39.189535 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.189504 2553 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 28 22:09:39.189535 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.189523 2553 kubelet.go:2448] "Starting kubelet main sync loop" Jul 28 22:09:39.189615 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:39.189556 2553 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 28 22:09:39.192151 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.192122 2553 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 22:09:39.217214 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.217190 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 22:09:39.218126 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.218109 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-132.ec2.internal" event="NodeHasSufficientMemory" Jul 28 22:09:39.218217 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.218138 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-132.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 22:09:39.218217 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.218148 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-132.ec2.internal" event="NodeHasSufficientPID" Jul 28 22:09:39.218217 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.218177 2553 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-132.ec2.internal" Jul 28 22:09:39.227802 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.227756 2553 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-132.ec2.internal" Jul 28 22:09:39.227802 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:39.227778 2553 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-132.ec2.internal\": node \"ip-10-0-142-132.ec2.internal\" not found" Jul 28 22:09:39.253942 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:39.253917 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-132.ec2.internal\" not found" Jul 28 22:09:39.290417 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.290389 2553 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-132.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-132.ec2.internal"] Jul 28 22:09:39.290496 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.290457 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 22:09:39.291381 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.291366 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-132.ec2.internal" event="NodeHasSufficientMemory" Jul 28 22:09:39.291481 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.291401 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-132.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 22:09:39.291481 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.291414 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-132.ec2.internal" event="NodeHasSufficientPID" Jul 28 22:09:39.292802 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.292787 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 22:09:39.292924 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.292907 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-132.ec2.internal" Jul 28 22:09:39.292972 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.292954 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 22:09:39.293556 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.293538 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-132.ec2.internal" event="NodeHasSufficientMemory" Jul 28 22:09:39.293621 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.293563 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-132.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 22:09:39.293621 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.293573 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-132.ec2.internal" event="NodeHasSufficientPID" Jul 28 22:09:39.293621 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.293538 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-132.ec2.internal" event="NodeHasSufficientMemory" Jul 28 22:09:39.293757 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.293652 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-132.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 22:09:39.293757 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.293673 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-132.ec2.internal" event="NodeHasSufficientPID" Jul 28 22:09:39.294720 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.294704 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-132.ec2.internal" Jul 28 22:09:39.294803 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.294733 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 22:09:39.295523 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.295508 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-132.ec2.internal" event="NodeHasSufficientMemory" Jul 28 22:09:39.295616 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.295537 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-132.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 22:09:39.295616 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.295552 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-132.ec2.internal" event="NodeHasSufficientPID" Jul 28 22:09:39.322458 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:39.322433 2553 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-132.ec2.internal\" not found" node="ip-10-0-142-132.ec2.internal" Jul 28 22:09:39.326989 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:39.326974 2553 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-132.ec2.internal\" not found" node="ip-10-0-142-132.ec2.internal" Jul 28 22:09:39.349880 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.349856 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a56b594b11650f501e0866607be01ff3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-132.ec2.internal\" (UID: \"a56b594b11650f501e0866607be01ff3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-132.ec2.internal" Jul 28 22:09:39.349947 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.349884 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a56b594b11650f501e0866607be01ff3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-132.ec2.internal\" (UID: \"a56b594b11650f501e0866607be01ff3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-132.ec2.internal" Jul 28 22:09:39.349947 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.349899 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/84e2d03b22c44b29314bad0b0a3a050e-config\") pod \"kube-apiserver-proxy-ip-10-0-142-132.ec2.internal\" (UID: \"84e2d03b22c44b29314bad0b0a3a050e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-132.ec2.internal" Jul 28 22:09:39.354474 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:39.354458 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-132.ec2.internal\" not found" Jul 28 22:09:39.450061 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.450028 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a56b594b11650f501e0866607be01ff3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-132.ec2.internal\" (UID: \"a56b594b11650f501e0866607be01ff3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-132.ec2.internal" Jul 28 22:09:39.450061 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.450042 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a56b594b11650f501e0866607be01ff3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-132.ec2.internal\" (UID: \"a56b594b11650f501e0866607be01ff3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-132.ec2.internal" Jul 28 22:09:39.450199 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.450087 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/84e2d03b22c44b29314bad0b0a3a050e-config\") pod \"kube-apiserver-proxy-ip-10-0-142-132.ec2.internal\" (UID: \"84e2d03b22c44b29314bad0b0a3a050e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-132.ec2.internal" Jul 28 22:09:39.450199 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.450106 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a56b594b11650f501e0866607be01ff3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-132.ec2.internal\" (UID: \"a56b594b11650f501e0866607be01ff3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-132.ec2.internal" Jul 28 22:09:39.450199 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.450132 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a56b594b11650f501e0866607be01ff3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-132.ec2.internal\" (UID: \"a56b594b11650f501e0866607be01ff3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-132.ec2.internal" Jul 28 22:09:39.450199 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.450170 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/84e2d03b22c44b29314bad0b0a3a050e-config\") pod \"kube-apiserver-proxy-ip-10-0-142-132.ec2.internal\" (UID: \"84e2d03b22c44b29314bad0b0a3a050e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-132.ec2.internal" Jul 28 22:09:39.455091 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:39.455071 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-132.ec2.internal\" not found" Jul 28 22:09:39.555911 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:39.555844 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-132.ec2.internal\" not found" Jul 28 22:09:39.625077 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.625047 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-132.ec2.internal" Jul 28 22:09:39.629784 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.629769 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-132.ec2.internal" Jul 28 22:09:39.656699 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:39.656672 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-132.ec2.internal\" not found" Jul 28 22:09:39.757185 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:39.757153 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-132.ec2.internal\" not found" Jul 28 22:09:39.857749 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:39.857675 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-132.ec2.internal\" not found" Jul 28 22:09:39.935402 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.935376 2553 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 22:09:39.958094 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:39.958070 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-132.ec2.internal\" not found" Jul 28 22:09:39.992247 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.992218 2553 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 28 22:09:39.992792 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.992352 2553 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 28 22:09:39.992792 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:39.992392 2553 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 28 22:09:41.267849 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:41.267816 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-132.ec2.internal\" not found" Jul 28 22:09:41.297773 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:41.297732 2553 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 22:04:40 +0000 UTC" deadline="2028-03-12 13:59:40.636679547 +0000 UTC" Jul 28 22:09:41.297773 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:41.297768 2553 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14223h49m59.338914468s" Jul 28 22:09:41.298832 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:41.298815 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 22:09:41.300067 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:41.300044 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-132.ec2.internal" event="NodeHasSufficientMemory" Jul 28 22:09:41.300160 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:41.300086 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-132.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 22:09:41.300160 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:41.300103 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-132.ec2.internal" event="NodeHasSufficientPID" Jul 28 22:09:41.313612 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:41.313584 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84e2d03b22c44b29314bad0b0a3a050e.slice/crio-c741358195ac98c91e4042f02caeff29e36325ab9ed16663db6ca28bce2cc7bd WatchSource:0}: Error finding container c741358195ac98c91e4042f02caeff29e36325ab9ed16663db6ca28bce2cc7bd: Status 404 returned error can't find the container with id c741358195ac98c91e4042f02caeff29e36325ab9ed16663db6ca28bce2cc7bd Jul 28 22:09:41.318098 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:41.318085 2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 22:09:41.364820 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:41.364795 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda56b594b11650f501e0866607be01ff3.slice/crio-77aa5dbb30c0a0bc1a86ea74ba1f144152aa3bc8d7b268b10c08092273ab26ef WatchSource:0}: Error finding container 77aa5dbb30c0a0bc1a86ea74ba1f144152aa3bc8d7b268b10c08092273ab26ef: Status 404 returned error can't find the container with id 77aa5dbb30c0a0bc1a86ea74ba1f144152aa3bc8d7b268b10c08092273ab26ef Jul 28 22:09:41.368823 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:41.368804 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-132.ec2.internal\" not found" Jul 28 22:09:41.401520 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:41.401476 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-132.ec2.internal" event={"ID":"a56b594b11650f501e0866607be01ff3","Type":"ContainerStarted","Data":"77aa5dbb30c0a0bc1a86ea74ba1f144152aa3bc8d7b268b10c08092273ab26ef"} Jul 28 22:09:41.402497 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:41.402472 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-132.ec2.internal" event={"ID":"84e2d03b22c44b29314bad0b0a3a050e","Type":"ContainerStarted","Data":"c741358195ac98c91e4042f02caeff29e36325ab9ed16663db6ca28bce2cc7bd"} Jul 28 22:09:41.456984 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:41.456959 2553 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 22:09:41.457900 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:41.457887 2553 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-132.ec2.internal" Jul 28 22:09:41.467101 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:41.467083 2553 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 28 22:09:41.468012 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:41.467997 2553 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-132.ec2.internal" Jul 28 22:09:41.479986 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:41.479964 2553 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 28 22:09:41.523371 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:41.523351 2553 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 22:09:42.236557 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.236511 2553 apiserver.go:52] "Watching apiserver" Jul 28 22:09:42.245284 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.245251 2553 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 28 22:09:42.246095 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.246065 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-7q942","kube-system/kube-apiserver-proxy-ip-10-0-142-132.ec2.internal","openshift-cluster-node-tuning-operator/tuned-6k2jz","openshift-dns/node-resolver-w9bnn","openshift-image-registry/node-ca-bs64q","openshift-multus/multus-xvmx5","openshift-network-diagnostics/network-check-target-qz6gw","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbzq5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-132.ec2.internal","openshift-multus/multus-additional-cni-plugins-zcxg2","openshift-multus/network-metrics-daemon-wl29s","openshift-network-operator/iptables-alerter-flt69","openshift-ovn-kubernetes/ovnkube-node-vzbxk","kube-system/global-pull-secret-syncer-djzl8"] Jul 28 22:09:42.247488 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.247469 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qz6gw" Jul 28 22:09:42.247581 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:42.247537 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qz6gw" podUID="3b595cc6-bee1-476f-842c-00ed98dd3e79" Jul 28 22:09:42.248830 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.248808 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" Jul 28 22:09:42.250342 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.250318 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6k2jz" Jul 28 22:09:42.251631 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.251345 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-w9bnn" Jul 28 22:09:42.251916 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.251894 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 28 22:09:42.252008 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.251932 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 28 22:09:42.252008 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.251944 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 28 22:09:42.252008 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.251960 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-dph6m\"" Jul 28 22:09:42.252163 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.251898 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 28 22:09:42.252163 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.251901 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 28 22:09:42.252163 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.251898 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 28 22:09:42.252725 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.252487 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ggdns\"" Jul 28 22:09:42.252725 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.252560 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 28 22:09:42.252725 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.252633 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 28 22:09:42.252725 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.252677 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bs64q" Jul 28 22:09:42.253457 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.253213 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 28 22:09:42.253948 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.253621 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xvmx5" Jul 28 22:09:42.253948 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.253695 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 28 22:09:42.253948 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.253737 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-gq5wl\"" Jul 28 22:09:42.255338 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.254758 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-djzl8" Jul 28 22:09:42.255338 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:42.254823 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-djzl8" podUID="59a1f798-88a8-4236-82f2-0a675d85920d" Jul 28 22:09:42.255338 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.255039 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 28 22:09:42.255338 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.255120 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-62km7\"" Jul 28 22:09:42.255338 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.255283 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 28 22:09:42.255338 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.255039 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 28 22:09:42.256069 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.255814 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 28 22:09:42.256069 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.255836 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-l9c5p\"" Jul 28 22:09:42.256069 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.255880 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 28 22:09:42.256069 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.255965 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 28 22:09:42.256069 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.256072 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 28 22:09:42.256334 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.256112 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7q942" Jul 28 22:09:42.257149 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.257129 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbzq5" Jul 28 22:09:42.257482 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.257283 2553 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 28 22:09:42.258277 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.258255 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 28 22:09:42.258411 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.258395 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 28 22:09:42.258517 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.258490 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-n4jnw\"" Jul 28 22:09:42.258924 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.258902 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 28 22:09:42.259040 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.259024 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 28 22:09:42.259116 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.259097 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 28 22:09:42.259370 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.259348 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wqphj\"" Jul 28 22:09:42.259440 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.259368 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zcxg2" Jul 28 22:09:42.261214 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.261198 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 28 22:09:42.261551 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.261532 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 28 22:09:42.261868 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.261853 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pzrlj\"" Jul 28 22:09:42.262058 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.262033 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wl29s" Jul 28 22:09:42.262135 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:42.262113 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wl29s" podUID="15e94249-e108-4426-b6f4-baf221c92a71" Jul 28 22:09:42.263457 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.263438 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-flt69" Jul 28 22:09:42.265550 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.265531 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 28 22:09:42.265901 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.265882 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 28 22:09:42.265977 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.265938 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 28 22:09:42.266064 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.266049 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-v48n8\"" Jul 28 22:09:42.266642 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.266620 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0f57ff96-5ed8-4a73-9dfd-7e5d45bd163c-hosts-file\") pod \"node-resolver-w9bnn\" (UID: \"0f57ff96-5ed8-4a73-9dfd-7e5d45bd163c\") " pod="openshift-dns/node-resolver-w9bnn" Jul 28 22:09:42.266750 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.266658 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/327f1776-1215-4c6e-9fef-3e30c12b1b66-cnibin\") pod \"multus-additional-cni-plugins-zcxg2\" (UID: \"327f1776-1215-4c6e-9fef-3e30c12b1b66\") " pod="openshift-multus/multus-additional-cni-plugins-zcxg2" Jul 28 22:09:42.266750 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.266689 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/842e457b-ef75-43bc-a182-e3abca66995a-host-cni-bin\") pod \"ovnkube-node-vzbxk\" (UID: \"842e457b-ef75-43bc-a182-e3abca66995a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" Jul 28 22:09:42.266750 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.266742 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/842e457b-ef75-43bc-a182-e3abca66995a-host-cni-netd\") pod \"ovnkube-node-vzbxk\" (UID: \"842e457b-ef75-43bc-a182-e3abca66995a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" Jul 28 22:09:42.266914 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.266757 2553 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 28 22:09:42.266914 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.266778 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/935749df-a4bd-4733-9085-1f8d1e4aa86a-etc-tuned\") pod \"tuned-6k2jz\" (UID: \"935749df-a4bd-4733-9085-1f8d1e4aa86a\") " pod="openshift-cluster-node-tuning-operator/tuned-6k2jz" Jul 28 22:09:42.266914 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.266803 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/74555be4-d49b-4d9c-9e80-c4604c40cd23-host-run-k8s-cni-cncf-io\") pod \"multus-xvmx5\" (UID: \"74555be4-d49b-4d9c-9e80-c4604c40cd23\") " pod="openshift-multus/multus-xvmx5" Jul 28 22:09:42.266914 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.266828 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/74555be4-d49b-4d9c-9e80-c4604c40cd23-multus-conf-dir\") pod \"multus-xvmx5\" (UID: \"74555be4-d49b-4d9c-9e80-c4604c40cd23\") " pod="openshift-multus/multus-xvmx5" Jul 28 22:09:42.266914 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.266854 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/327f1776-1215-4c6e-9fef-3e30c12b1b66-system-cni-dir\") pod \"multus-additional-cni-plugins-zcxg2\" (UID: \"327f1776-1215-4c6e-9fef-3e30c12b1b66\") " pod="openshift-multus/multus-additional-cni-plugins-zcxg2" Jul 28 22:09:42.266914 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.266890 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/54dc150c-a1f7-4df4-9adf-03077bf59073-host\") pod \"node-ca-bs64q\" (UID: \"54dc150c-a1f7-4df4-9adf-03077bf59073\") " pod="openshift-image-registry/node-ca-bs64q" Jul 28 22:09:42.266914 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.266912 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/842e457b-ef75-43bc-a182-e3abca66995a-node-log\") pod \"ovnkube-node-vzbxk\" (UID: \"842e457b-ef75-43bc-a182-e3abca66995a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" Jul 28 22:09:42.267190 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.266945 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/935749df-a4bd-4733-9085-1f8d1e4aa86a-etc-sysctl-conf\") pod \"tuned-6k2jz\" (UID: \"935749df-a4bd-4733-9085-1f8d1e4aa86a\") " pod="openshift-cluster-node-tuning-operator/tuned-6k2jz" Jul 28 22:09:42.267190 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.266972 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/74555be4-d49b-4d9c-9e80-c4604c40cd23-host-var-lib-cni-bin\") pod \"multus-xvmx5\" (UID: \"74555be4-d49b-4d9c-9e80-c4604c40cd23\") " pod="openshift-multus/multus-xvmx5" Jul 28 22:09:42.267190 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.266999 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/74555be4-d49b-4d9c-9e80-c4604c40cd23-multus-daemon-config\") pod \"multus-xvmx5\" (UID: \"74555be4-d49b-4d9c-9e80-c4604c40cd23\") " pod="openshift-multus/multus-xvmx5" Jul 28 22:09:42.267190 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.267026 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/327f1776-1215-4c6e-9fef-3e30c12b1b66-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zcxg2\" (UID: \"327f1776-1215-4c6e-9fef-3e30c12b1b66\") " pod="openshift-multus/multus-additional-cni-plugins-zcxg2" Jul 28 22:09:42.267190 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.267056 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a55216ee-cd69-4f12-af05-1d12a69e7c93-device-dir\") pod \"aws-ebs-csi-driver-node-fbzq5\" (UID: \"a55216ee-cd69-4f12-af05-1d12a69e7c93\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbzq5" Jul 28 22:09:42.267190 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.267114 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/842e457b-ef75-43bc-a182-e3abca66995a-run-ovn\") pod \"ovnkube-node-vzbxk\" (UID: \"842e457b-ef75-43bc-a182-e3abca66995a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" Jul 28 22:09:42.267190 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.267146 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/842e457b-ef75-43bc-a182-e3abca66995a-host-run-ovn-kubernetes\") pod \"ovnkube-node-vzbxk\" (UID: \"842e457b-ef75-43bc-a182-e3abca66995a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" Jul 28 22:09:42.267190 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.267175 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/59a1f798-88a8-4236-82f2-0a675d85920d-kubelet-config\") pod \"global-pull-secret-syncer-djzl8\" (UID: \"59a1f798-88a8-4236-82f2-0a675d85920d\") " pod="kube-system/global-pull-secret-syncer-djzl8" Jul 28 22:09:42.267555 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.267200 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/59a1f798-88a8-4236-82f2-0a675d85920d-dbus\") pod \"global-pull-secret-syncer-djzl8\" (UID: \"59a1f798-88a8-4236-82f2-0a675d85920d\") " pod="kube-system/global-pull-secret-syncer-djzl8" Jul 28 22:09:42.267555 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.267235 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/935749df-a4bd-4733-9085-1f8d1e4aa86a-host\") pod \"tuned-6k2jz\" (UID: \"935749df-a4bd-4733-9085-1f8d1e4aa86a\") " pod="openshift-cluster-node-tuning-operator/tuned-6k2jz" Jul 28 22:09:42.267555 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.267273 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/74555be4-d49b-4d9c-9e80-c4604c40cd23-host-run-netns\") pod \"multus-xvmx5\" (UID: \"74555be4-d49b-4d9c-9e80-c4604c40cd23\") " pod="openshift-multus/multus-xvmx5" Jul 28 22:09:42.267555 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.267309 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/842e457b-ef75-43bc-a182-e3abca66995a-host-kubelet\") pod \"ovnkube-node-vzbxk\" (UID: \"842e457b-ef75-43bc-a182-e3abca66995a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" Jul 28 22:09:42.267555 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.267331 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/842e457b-ef75-43bc-a182-e3abca66995a-log-socket\") pod \"ovnkube-node-vzbxk\" (UID: \"842e457b-ef75-43bc-a182-e3abca66995a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" Jul 28 22:09:42.267555 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.267354 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27hqt\" (UniqueName: \"kubernetes.io/projected/842e457b-ef75-43bc-a182-e3abca66995a-kube-api-access-27hqt\") pod \"ovnkube-node-vzbxk\" (UID: \"842e457b-ef75-43bc-a182-e3abca66995a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" Jul 28 22:09:42.267555 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.267378 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/935749df-a4bd-4733-9085-1f8d1e4aa86a-etc-modprobe-d\") pod \"tuned-6k2jz\" (UID: \"935749df-a4bd-4733-9085-1f8d1e4aa86a\") " pod="openshift-cluster-node-tuning-operator/tuned-6k2jz" Jul 28 22:09:42.267555 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.267401 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/935749df-a4bd-4733-9085-1f8d1e4aa86a-etc-kubernetes\") pod \"tuned-6k2jz\" (UID: \"935749df-a4bd-4733-9085-1f8d1e4aa86a\") " pod="openshift-cluster-node-tuning-operator/tuned-6k2jz" Jul 28 22:09:42.267555 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.267439 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/935749df-a4bd-4733-9085-1f8d1e4aa86a-etc-systemd\") pod \"tuned-6k2jz\" (UID: \"935749df-a4bd-4733-9085-1f8d1e4aa86a\") " pod="openshift-cluster-node-tuning-operator/tuned-6k2jz" Jul 28 22:09:42.267555 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.267484 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/935749df-a4bd-4733-9085-1f8d1e4aa86a-var-lib-kubelet\") pod \"tuned-6k2jz\" (UID: \"935749df-a4bd-4733-9085-1f8d1e4aa86a\") " pod="openshift-cluster-node-tuning-operator/tuned-6k2jz" Jul 28 22:09:42.267555 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.267508 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/842e457b-ef75-43bc-a182-e3abca66995a-ovn-node-metrics-cert\") pod \"ovnkube-node-vzbxk\" (UID: \"842e457b-ef75-43bc-a182-e3abca66995a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" Jul 28 22:09:42.267555 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.267545 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/74555be4-d49b-4d9c-9e80-c4604c40cd23-cni-binary-copy\") pod \"multus-xvmx5\" (UID: \"74555be4-d49b-4d9c-9e80-c4604c40cd23\") " pod="openshift-multus/multus-xvmx5" Jul 28 22:09:42.268153 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.267580 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/74555be4-d49b-4d9c-9e80-c4604c40cd23-host-var-lib-kubelet\") pod \"multus-xvmx5\" (UID: \"74555be4-d49b-4d9c-9e80-c4604c40cd23\") " pod="openshift-multus/multus-xvmx5" Jul 28 22:09:42.268153 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.267634 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/74555be4-d49b-4d9c-9e80-c4604c40cd23-hostroot\") pod \"multus-xvmx5\" (UID: \"74555be4-d49b-4d9c-9e80-c4604c40cd23\") " pod="openshift-multus/multus-xvmx5" Jul 28 22:09:42.268153 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.267653 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/74555be4-d49b-4d9c-9e80-c4604c40cd23-host-run-multus-certs\") pod \"multus-xvmx5\" (UID: \"74555be4-d49b-4d9c-9e80-c4604c40cd23\") " pod="openshift-multus/multus-xvmx5" Jul 28 22:09:42.268153 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.267676 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b678w\" (UniqueName: \"kubernetes.io/projected/327f1776-1215-4c6e-9fef-3e30c12b1b66-kube-api-access-b678w\") pod \"multus-additional-cni-plugins-zcxg2\" (UID: \"327f1776-1215-4c6e-9fef-3e30c12b1b66\") " pod="openshift-multus/multus-additional-cni-plugins-zcxg2" Jul 28 22:09:42.268153 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.267735 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/59a1f798-88a8-4236-82f2-0a675d85920d-original-pull-secret\") pod \"global-pull-secret-syncer-djzl8\" (UID: \"59a1f798-88a8-4236-82f2-0a675d85920d\") " pod="kube-system/global-pull-secret-syncer-djzl8" Jul 28 22:09:42.268153 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.267760 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a55216ee-cd69-4f12-af05-1d12a69e7c93-registration-dir\") pod \"aws-ebs-csi-driver-node-fbzq5\" (UID: \"a55216ee-cd69-4f12-af05-1d12a69e7c93\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbzq5" Jul 28 22:09:42.268153 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.267783 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/842e457b-ef75-43bc-a182-e3abca66995a-run-openvswitch\") pod \"ovnkube-node-vzbxk\" (UID: \"842e457b-ef75-43bc-a182-e3abca66995a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" Jul 28 22:09:42.268153 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.267811 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/842e457b-ef75-43bc-a182-e3abca66995a-ovnkube-config\") pod \"ovnkube-node-vzbxk\" (UID: \"842e457b-ef75-43bc-a182-e3abca66995a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" Jul 28 22:09:42.268153 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.267862 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/74555be4-d49b-4d9c-9e80-c4604c40cd23-multus-cni-dir\") pod \"multus-xvmx5\" (UID: \"74555be4-d49b-4d9c-9e80-c4604c40cd23\") " pod="openshift-multus/multus-xvmx5" Jul 28 22:09:42.268153 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.267905 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/327f1776-1215-4c6e-9fef-3e30c12b1b66-os-release\") pod \"multus-additional-cni-plugins-zcxg2\" (UID: \"327f1776-1215-4c6e-9fef-3e30c12b1b66\") " pod="openshift-multus/multus-additional-cni-plugins-zcxg2" Jul 28 22:09:42.268153 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.267929 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/327f1776-1215-4c6e-9fef-3e30c12b1b66-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zcxg2\" (UID: \"327f1776-1215-4c6e-9fef-3e30c12b1b66\") " pod="openshift-multus/multus-additional-cni-plugins-zcxg2" Jul 28 22:09:42.268153 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.267961 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/54dc150c-a1f7-4df4-9adf-03077bf59073-serviceca\") pod \"node-ca-bs64q\" (UID: \"54dc150c-a1f7-4df4-9adf-03077bf59073\") " pod="openshift-image-registry/node-ca-bs64q" Jul 28 22:09:42.268153 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.268015 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/842e457b-ef75-43bc-a182-e3abca66995a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vzbxk\" (UID: \"842e457b-ef75-43bc-a182-e3abca66995a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" Jul 28 22:09:42.268153 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.268047 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f57ff96-5ed8-4a73-9dfd-7e5d45bd163c-tmp-dir\") pod \"node-resolver-w9bnn\" (UID: \"0f57ff96-5ed8-4a73-9dfd-7e5d45bd163c\") " pod="openshift-dns/node-resolver-w9bnn" Jul 28 22:09:42.268741 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.268071 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c93df2e7-d79a-4a98-8556-a98868dfd93e-konnectivity-ca\") pod \"konnectivity-agent-7q942\" (UID: \"c93df2e7-d79a-4a98-8556-a98868dfd93e\") " pod="kube-system/konnectivity-agent-7q942" Jul 28 22:09:42.268860 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.268808 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a55216ee-cd69-4f12-af05-1d12a69e7c93-socket-dir\") pod \"aws-ebs-csi-driver-node-fbzq5\" (UID: \"a55216ee-cd69-4f12-af05-1d12a69e7c93\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbzq5" Jul 28 22:09:42.268912 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.268892 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kthfk\" (UniqueName: \"kubernetes.io/projected/a55216ee-cd69-4f12-af05-1d12a69e7c93-kube-api-access-kthfk\") pod \"aws-ebs-csi-driver-node-fbzq5\" (UID: \"a55216ee-cd69-4f12-af05-1d12a69e7c93\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbzq5" Jul 28 22:09:42.268956 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.268931 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/842e457b-ef75-43bc-a182-e3abca66995a-host-run-netns\") pod \"ovnkube-node-vzbxk\" (UID: \"842e457b-ef75-43bc-a182-e3abca66995a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" Jul 28 22:09:42.269005 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.268965 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/935749df-a4bd-4733-9085-1f8d1e4aa86a-etc-sysctl-d\") pod \"tuned-6k2jz\" (UID: \"935749df-a4bd-4733-9085-1f8d1e4aa86a\") " pod="openshift-cluster-node-tuning-operator/tuned-6k2jz" Jul 28 22:09:42.269053 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.269003 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/74555be4-d49b-4d9c-9e80-c4604c40cd23-cnibin\") pod \"multus-xvmx5\" (UID: \"74555be4-d49b-4d9c-9e80-c4604c40cd23\") " pod="openshift-multus/multus-xvmx5" Jul 28 22:09:42.269105 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.269048 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/74555be4-d49b-4d9c-9e80-c4604c40cd23-multus-socket-dir-parent\") pod \"multus-xvmx5\" (UID: \"74555be4-d49b-4d9c-9e80-c4604c40cd23\") " pod="openshift-multus/multus-xvmx5" Jul 28 22:09:42.269105 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.269077 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5cns\" (UniqueName: \"kubernetes.io/projected/74555be4-d49b-4d9c-9e80-c4604c40cd23-kube-api-access-j5cns\") pod \"multus-xvmx5\" (UID: \"74555be4-d49b-4d9c-9e80-c4604c40cd23\") " pod="openshift-multus/multus-xvmx5" Jul 28 22:09:42.269197 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.269131 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/327f1776-1215-4c6e-9fef-3e30c12b1b66-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zcxg2\" (UID: \"327f1776-1215-4c6e-9fef-3e30c12b1b66\") " pod="openshift-multus/multus-additional-cni-plugins-zcxg2" Jul 28 22:09:42.269197 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.269163 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/935749df-a4bd-4733-9085-1f8d1e4aa86a-tmp\") pod \"tuned-6k2jz\" (UID: \"935749df-a4bd-4733-9085-1f8d1e4aa86a\") " pod="openshift-cluster-node-tuning-operator/tuned-6k2jz" Jul 28 22:09:42.269197 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.269192 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgpqd\" (UniqueName: \"kubernetes.io/projected/0f57ff96-5ed8-4a73-9dfd-7e5d45bd163c-kube-api-access-kgpqd\") pod \"node-resolver-w9bnn\" (UID: \"0f57ff96-5ed8-4a73-9dfd-7e5d45bd163c\") " pod="openshift-dns/node-resolver-w9bnn" Jul 28 22:09:42.269352 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.269217 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/842e457b-ef75-43bc-a182-e3abca66995a-etc-openvswitch\") pod \"ovnkube-node-vzbxk\" (UID: \"842e457b-ef75-43bc-a182-e3abca66995a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" Jul 28 22:09:42.269352 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.269232 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/74555be4-d49b-4d9c-9e80-c4604c40cd23-system-cni-dir\") pod \"multus-xvmx5\" (UID: \"74555be4-d49b-4d9c-9e80-c4604c40cd23\") " pod="openshift-multus/multus-xvmx5" Jul 28 22:09:42.269352 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.269265 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m97d\" (UniqueName: \"kubernetes.io/projected/3b595cc6-bee1-476f-842c-00ed98dd3e79-kube-api-access-8m97d\") pod \"network-check-target-qz6gw\" (UID: \"3b595cc6-bee1-476f-842c-00ed98dd3e79\") " pod="openshift-network-diagnostics/network-check-target-qz6gw" Jul 28 22:09:42.269352 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.269306 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/842e457b-ef75-43bc-a182-e3abca66995a-host-slash\") pod \"ovnkube-node-vzbxk\" (UID: \"842e457b-ef75-43bc-a182-e3abca66995a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" Jul 28 22:09:42.269352 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.269340 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/842e457b-ef75-43bc-a182-e3abca66995a-run-systemd\") pod \"ovnkube-node-vzbxk\" (UID: \"842e457b-ef75-43bc-a182-e3abca66995a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" Jul 28 22:09:42.269573 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.269365 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/842e457b-ef75-43bc-a182-e3abca66995a-var-lib-openvswitch\") pod \"ovnkube-node-vzbxk\" (UID: \"842e457b-ef75-43bc-a182-e3abca66995a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" Jul 28 22:09:42.269573 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.269394 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/935749df-a4bd-4733-9085-1f8d1e4aa86a-etc-sysconfig\") pod \"tuned-6k2jz\" (UID: \"935749df-a4bd-4733-9085-1f8d1e4aa86a\") " pod="openshift-cluster-node-tuning-operator/tuned-6k2jz" Jul 28 22:09:42.269573 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.269415 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/935749df-a4bd-4733-9085-1f8d1e4aa86a-sys\") pod \"tuned-6k2jz\" (UID: \"935749df-a4bd-4733-9085-1f8d1e4aa86a\") " pod="openshift-cluster-node-tuning-operator/tuned-6k2jz" Jul 28 22:09:42.269573 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.269435 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/935749df-a4bd-4733-9085-1f8d1e4aa86a-run\") pod \"tuned-6k2jz\" (UID: \"935749df-a4bd-4733-9085-1f8d1e4aa86a\") " pod="openshift-cluster-node-tuning-operator/tuned-6k2jz" Jul 28 22:09:42.269573 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.269460 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/74555be4-d49b-4d9c-9e80-c4604c40cd23-host-var-lib-cni-multus\") pod \"multus-xvmx5\" (UID: \"74555be4-d49b-4d9c-9e80-c4604c40cd23\") " pod="openshift-multus/multus-xvmx5" Jul 28 22:09:42.269573 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.269488 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/74555be4-d49b-4d9c-9e80-c4604c40cd23-etc-kubernetes\") pod \"multus-xvmx5\" (UID: \"74555be4-d49b-4d9c-9e80-c4604c40cd23\") " pod="openshift-multus/multus-xvmx5" Jul 28 22:09:42.269573 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.269509 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c93df2e7-d79a-4a98-8556-a98868dfd93e-agent-certs\") pod \"konnectivity-agent-7q942\" (UID: \"c93df2e7-d79a-4a98-8556-a98868dfd93e\") " pod="kube-system/konnectivity-agent-7q942" Jul 28 22:09:42.269573 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.269528 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a55216ee-cd69-4f12-af05-1d12a69e7c93-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fbzq5\" (UID: \"a55216ee-cd69-4f12-af05-1d12a69e7c93\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbzq5" Jul 28 22:09:42.269573 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.269556 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/842e457b-ef75-43bc-a182-e3abca66995a-env-overrides\") pod \"ovnkube-node-vzbxk\" (UID: \"842e457b-ef75-43bc-a182-e3abca66995a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" Jul 28 22:09:42.269990 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.269577 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/327f1776-1215-4c6e-9fef-3e30c12b1b66-cni-binary-copy\") pod \"multus-additional-cni-plugins-zcxg2\" (UID: \"327f1776-1215-4c6e-9fef-3e30c12b1b66\") " pod="openshift-multus/multus-additional-cni-plugins-zcxg2" Jul 28 22:09:42.269990 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.269600 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx25l\" (UniqueName: \"kubernetes.io/projected/54dc150c-a1f7-4df4-9adf-03077bf59073-kube-api-access-hx25l\") pod \"node-ca-bs64q\" (UID: \"54dc150c-a1f7-4df4-9adf-03077bf59073\") " pod="openshift-image-registry/node-ca-bs64q" Jul 28 22:09:42.269990 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.269616 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a55216ee-cd69-4f12-af05-1d12a69e7c93-sys-fs\") pod \"aws-ebs-csi-driver-node-fbzq5\" (UID: \"a55216ee-cd69-4f12-af05-1d12a69e7c93\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbzq5" Jul 28 22:09:42.269990 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.269637 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/842e457b-ef75-43bc-a182-e3abca66995a-ovnkube-script-lib\") pod \"ovnkube-node-vzbxk\" (UID: \"842e457b-ef75-43bc-a182-e3abca66995a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" Jul 28 22:09:42.269990 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.269665 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8czw2\" (UniqueName: \"kubernetes.io/projected/935749df-a4bd-4733-9085-1f8d1e4aa86a-kube-api-access-8czw2\") pod \"tuned-6k2jz\" (UID: \"935749df-a4bd-4733-9085-1f8d1e4aa86a\") " pod="openshift-cluster-node-tuning-operator/tuned-6k2jz" Jul 28 22:09:42.269990 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.269684 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/74555be4-d49b-4d9c-9e80-c4604c40cd23-os-release\") pod \"multus-xvmx5\" (UID: \"74555be4-d49b-4d9c-9e80-c4604c40cd23\") " pod="openshift-multus/multus-xvmx5" Jul 28 22:09:42.269990 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.269727 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a55216ee-cd69-4f12-af05-1d12a69e7c93-etc-selinux\") pod \"aws-ebs-csi-driver-node-fbzq5\" (UID: \"a55216ee-cd69-4f12-af05-1d12a69e7c93\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbzq5" Jul 28 22:09:42.269990 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.269749 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/842e457b-ef75-43bc-a182-e3abca66995a-systemd-units\") pod \"ovnkube-node-vzbxk\" (UID: \"842e457b-ef75-43bc-a182-e3abca66995a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" Jul 28 22:09:42.269990 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.269775 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/935749df-a4bd-4733-9085-1f8d1e4aa86a-lib-modules\") pod \"tuned-6k2jz\" (UID: \"935749df-a4bd-4733-9085-1f8d1e4aa86a\") " pod="openshift-cluster-node-tuning-operator/tuned-6k2jz" Jul 28 22:09:42.287235 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.287200 2553 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-s2hkh" Jul 28 22:09:42.295097 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.295073 2553 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-s2hkh" Jul 28 22:09:42.359049 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.359012 2553 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 28 22:09:42.370190 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.370153 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/935749df-a4bd-4733-9085-1f8d1e4aa86a-tmp\") pod \"tuned-6k2jz\" (UID: \"935749df-a4bd-4733-9085-1f8d1e4aa86a\") " pod="openshift-cluster-node-tuning-operator/tuned-6k2jz" Jul 28 22:09:42.370190 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.370189 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgpqd\" (UniqueName: \"kubernetes.io/projected/0f57ff96-5ed8-4a73-9dfd-7e5d45bd163c-kube-api-access-kgpqd\") pod \"node-resolver-w9bnn\" (UID: \"0f57ff96-5ed8-4a73-9dfd-7e5d45bd163c\") " pod="openshift-dns/node-resolver-w9bnn" Jul 28 22:09:42.370408 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.370214 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwkwh\" (UniqueName: \"kubernetes.io/projected/cb4e193d-e952-43fb-9161-f45d39dd37ec-kube-api-access-wwkwh\") pod \"iptables-alerter-flt69\" (UID: \"cb4e193d-e952-43fb-9161-f45d39dd37ec\") " pod="openshift-network-operator/iptables-alerter-flt69" Jul 28 22:09:42.370408 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.370231 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/842e457b-ef75-43bc-a182-e3abca66995a-etc-openvswitch\") pod \"ovnkube-node-vzbxk\" (UID: \"842e457b-ef75-43bc-a182-e3abca66995a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" Jul 28 22:09:42.370408 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.370246 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/74555be4-d49b-4d9c-9e80-c4604c40cd23-system-cni-dir\") pod \"multus-xvmx5\" (UID: \"74555be4-d49b-4d9c-9e80-c4604c40cd23\") " pod="openshift-multus/multus-xvmx5" Jul 28 22:09:42.370408 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.370264 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m97d\" (UniqueName: \"kubernetes.io/projected/3b595cc6-bee1-476f-842c-00ed98dd3e79-kube-api-access-8m97d\") pod \"network-check-target-qz6gw\" (UID: \"3b595cc6-bee1-476f-842c-00ed98dd3e79\") " pod="openshift-network-diagnostics/network-check-target-qz6gw" Jul 28 22:09:42.370408 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.370287 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/842e457b-ef75-43bc-a182-e3abca66995a-host-slash\") pod \"ovnkube-node-vzbxk\" (UID: \"842e457b-ef75-43bc-a182-e3abca66995a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" Jul 28 22:09:42.370408 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.370324 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/842e457b-ef75-43bc-a182-e3abca66995a-run-systemd\") pod \"ovnkube-node-vzbxk\" (UID: \"842e457b-ef75-43bc-a182-e3abca66995a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" Jul 28 22:09:42.370408 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.370324 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/842e457b-ef75-43bc-a182-e3abca66995a-etc-openvswitch\") pod \"ovnkube-node-vzbxk\" (UID: \"842e457b-ef75-43bc-a182-e3abca66995a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" Jul 28 22:09:42.370408 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.370333 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/74555be4-d49b-4d9c-9e80-c4604c40cd23-system-cni-dir\") pod \"multus-xvmx5\" (UID: \"74555be4-d49b-4d9c-9e80-c4604c40cd23\") " pod="openshift-multus/multus-xvmx5" Jul 28 22:09:42.370408 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.370369 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/842e457b-ef75-43bc-a182-e3abca66995a-host-slash\") pod \"ovnkube-node-vzbxk\" (UID: \"842e457b-ef75-43bc-a182-e3abca66995a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" Jul 28 22:09:42.370408 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.370396 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/842e457b-ef75-43bc-a182-e3abca66995a-run-systemd\") pod \"ovnkube-node-vzbxk\" (UID: \"842e457b-ef75-43bc-a182-e3abca66995a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" Jul 28 22:09:42.370920 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.370486 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/842e457b-ef75-43bc-a182-e3abca66995a-var-lib-openvswitch\") pod \"ovnkube-node-vzbxk\" (UID: \"842e457b-ef75-43bc-a182-e3abca66995a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" Jul 28 22:09:42.370920 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.370519 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/935749df-a4bd-4733-9085-1f8d1e4aa86a-etc-sysconfig\") pod \"tuned-6k2jz\" (UID: \"935749df-a4bd-4733-9085-1f8d1e4aa86a\") " pod="openshift-cluster-node-tuning-operator/tuned-6k2jz" Jul 28 22:09:42.370920 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.370522 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/842e457b-ef75-43bc-a182-e3abca66995a-var-lib-openvswitch\") pod \"ovnkube-node-vzbxk\" (UID: \"842e457b-ef75-43bc-a182-e3abca66995a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" Jul 28 22:09:42.370920 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.370562 2553 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 28 22:09:42.370920 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.370577 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/935749df-a4bd-4733-9085-1f8d1e4aa86a-sys\") pod \"tuned-6k2jz\" (UID: \"935749df-a4bd-4733-9085-1f8d1e4aa86a\") " pod="openshift-cluster-node-tuning-operator/tuned-6k2jz" Jul 28 22:09:42.370920 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.370600 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15e94249-e108-4426-b6f4-baf221c92a71-metrics-certs\") pod \"network-metrics-daemon-wl29s\" (UID: \"15e94249-e108-4426-b6f4-baf221c92a71\") " pod="openshift-multus/network-metrics-daemon-wl29s" Jul 28 22:09:42.370920 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.370628 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cb4e193d-e952-43fb-9161-f45d39dd37ec-host-slash\") pod \"iptables-alerter-flt69\" (UID: \"cb4e193d-e952-43fb-9161-f45d39dd37ec\") " pod="openshift-network-operator/iptables-alerter-flt69" Jul 28 22:09:42.370920 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.370655 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/935749df-a4bd-4733-9085-1f8d1e4aa86a-sys\") pod \"tuned-6k2jz\" (UID: \"935749df-a4bd-4733-9085-1f8d1e4aa86a\") " pod="openshift-cluster-node-tuning-operator/tuned-6k2jz" Jul 28 22:09:42.370920 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.370670 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/935749df-a4bd-4733-9085-1f8d1e4aa86a-etc-sysconfig\") pod \"tuned-6k2jz\" (UID: \"935749df-a4bd-4733-9085-1f8d1e4aa86a\") " pod="openshift-cluster-node-tuning-operator/tuned-6k2jz" Jul 28 22:09:42.370920 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.370684 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/935749df-a4bd-4733-9085-1f8d1e4aa86a-run\") pod \"tuned-6k2jz\" (UID: \"935749df-a4bd-4733-9085-1f8d1e4aa86a\") " pod="openshift-cluster-node-tuning-operator/tuned-6k2jz" Jul 28 22:09:42.370920 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.370742 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/935749df-a4bd-4733-9085-1f8d1e4aa86a-run\") pod \"tuned-6k2jz\" (UID: \"935749df-a4bd-4733-9085-1f8d1e4aa86a\") " pod="openshift-cluster-node-tuning-operator/tuned-6k2jz" Jul 28 22:09:42.370920 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.370758 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/74555be4-d49b-4d9c-9e80-c4604c40cd23-host-var-lib-cni-multus\") pod \"multus-xvmx5\" (UID: \"74555be4-d49b-4d9c-9e80-c4604c40cd23\") " pod="openshift-multus/multus-xvmx5" Jul 28 22:09:42.370920 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.370785 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/74555be4-d49b-4d9c-9e80-c4604c40cd23-etc-kubernetes\") pod \"multus-xvmx5\" (UID: \"74555be4-d49b-4d9c-9e80-c4604c40cd23\") " pod="openshift-multus/multus-xvmx5" Jul 28 22:09:42.370920 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.370819 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c93df2e7-d79a-4a98-8556-a98868dfd93e-agent-certs\") pod \"konnectivity-agent-7q942\" (UID: \"c93df2e7-d79a-4a98-8556-a98868dfd93e\") " pod="kube-system/konnectivity-agent-7q942" Jul 28 22:09:42.370920 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.370844 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a55216ee-cd69-4f12-af05-1d12a69e7c93-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fbzq5\" (UID: \"a55216ee-cd69-4f12-af05-1d12a69e7c93\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbzq5" Jul 28 22:09:42.370920 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.370847 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/74555be4-d49b-4d9c-9e80-c4604c40cd23-host-var-lib-cni-multus\") pod \"multus-xvmx5\" (UID: \"74555be4-d49b-4d9c-9e80-c4604c40cd23\") " pod="openshift-multus/multus-xvmx5" Jul 28 22:09:42.370920 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.370856 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/74555be4-d49b-4d9c-9e80-c4604c40cd23-etc-kubernetes\") pod \"multus-xvmx5\" (UID: \"74555be4-d49b-4d9c-9e80-c4604c40cd23\") " pod="openshift-multus/multus-xvmx5" Jul 28 22:09:42.370920 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.370867 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/842e457b-ef75-43bc-a182-e3abca66995a-env-overrides\") pod \"ovnkube-node-vzbxk\" (UID: \"842e457b-ef75-43bc-a182-e3abca66995a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" Jul 28 22:09:42.371804 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.370891 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/327f1776-1215-4c6e-9fef-3e30c12b1b66-cni-binary-copy\") pod \"multus-additional-cni-plugins-zcxg2\" (UID: \"327f1776-1215-4c6e-9fef-3e30c12b1b66\") " pod="openshift-multus/multus-additional-cni-plugins-zcxg2" Jul 28 22:09:42.371804 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.370912 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a55216ee-cd69-4f12-af05-1d12a69e7c93-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fbzq5\" (UID: \"a55216ee-cd69-4f12-af05-1d12a69e7c93\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbzq5" Jul 28 22:09:42.371804 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.370916 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx25l\" (UniqueName: \"kubernetes.io/projected/54dc150c-a1f7-4df4-9adf-03077bf59073-kube-api-access-hx25l\") pod \"node-ca-bs64q\" (UID: \"54dc150c-a1f7-4df4-9adf-03077bf59073\") " pod="openshift-image-registry/node-ca-bs64q" Jul 28 22:09:42.371804 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.370952 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a55216ee-cd69-4f12-af05-1d12a69e7c93-sys-fs\") pod \"aws-ebs-csi-driver-node-fbzq5\" (UID: \"a55216ee-cd69-4f12-af05-1d12a69e7c93\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbzq5" Jul 28 22:09:42.371804 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.370978 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/842e457b-ef75-43bc-a182-e3abca66995a-ovnkube-script-lib\") pod \"ovnkube-node-vzbxk\" (UID: \"842e457b-ef75-43bc-a182-e3abca66995a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" Jul 28 22:09:42.371804 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.371030 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8czw2\" (UniqueName: \"kubernetes.io/projected/935749df-a4bd-4733-9085-1f8d1e4aa86a-kube-api-access-8czw2\") pod \"tuned-6k2jz\" (UID: \"935749df-a4bd-4733-9085-1f8d1e4aa86a\") " pod="openshift-cluster-node-tuning-operator/tuned-6k2jz" Jul 28 22:09:42.371804 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.371074 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/74555be4-d49b-4d9c-9e80-c4604c40cd23-os-release\") pod \"multus-xvmx5\" (UID: \"74555be4-d49b-4d9c-9e80-c4604c40cd23\") " pod="openshift-multus/multus-xvmx5" Jul 28 22:09:42.371804 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.371099 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a55216ee-cd69-4f12-af05-1d12a69e7c93-etc-selinux\") pod \"aws-ebs-csi-driver-node-fbzq5\" (UID: \"a55216ee-cd69-4f12-af05-1d12a69e7c93\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbzq5" Jul 28 22:09:42.371804 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.371121 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/842e457b-ef75-43bc-a182-e3abca66995a-systemd-units\") pod \"ovnkube-node-vzbxk\" (UID: \"842e457b-ef75-43bc-a182-e3abca66995a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" Jul 28 22:09:42.371804 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.371143 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/935749df-a4bd-4733-9085-1f8d1e4aa86a-lib-modules\") pod \"tuned-6k2jz\" (UID: \"935749df-a4bd-4733-9085-1f8d1e4aa86a\") " pod="openshift-cluster-node-tuning-operator/tuned-6k2jz" Jul 28 22:09:42.371804 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.371167 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0f57ff96-5ed8-4a73-9dfd-7e5d45bd163c-hosts-file\") pod \"node-resolver-w9bnn\" (UID: \"0f57ff96-5ed8-4a73-9dfd-7e5d45bd163c\") " pod="openshift-dns/node-resolver-w9bnn" Jul 28 22:09:42.371804 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.371191 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/327f1776-1215-4c6e-9fef-3e30c12b1b66-cnibin\") pod \"multus-additional-cni-plugins-zcxg2\" (UID: \"327f1776-1215-4c6e-9fef-3e30c12b1b66\") " pod="openshift-multus/multus-additional-cni-plugins-zcxg2" Jul 28 22:09:42.371804 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.371215 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/842e457b-ef75-43bc-a182-e3abca66995a-host-cni-bin\") pod \"ovnkube-node-vzbxk\" (UID: \"842e457b-ef75-43bc-a182-e3abca66995a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" Jul 28 22:09:42.371804 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.371274 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/842e457b-ef75-43bc-a182-e3abca66995a-host-cni-netd\") pod \"ovnkube-node-vzbxk\" (UID: \"842e457b-ef75-43bc-a182-e3abca66995a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" Jul 28 22:09:42.371804 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.371300 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/935749df-a4bd-4733-9085-1f8d1e4aa86a-etc-tuned\") pod \"tuned-6k2jz\" (UID: \"935749df-a4bd-4733-9085-1f8d1e4aa86a\") " pod="openshift-cluster-node-tuning-operator/tuned-6k2jz" Jul 28 22:09:42.371804 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.371322 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/74555be4-d49b-4d9c-9e80-c4604c40cd23-host-run-k8s-cni-cncf-io\") pod \"multus-xvmx5\" (UID: \"74555be4-d49b-4d9c-9e80-c4604c40cd23\") " pod="openshift-multus/multus-xvmx5" Jul 28 22:09:42.371804 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.371325 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a55216ee-cd69-4f12-af05-1d12a69e7c93-etc-selinux\") pod \"aws-ebs-csi-driver-node-fbzq5\" (UID: \"a55216ee-cd69-4f12-af05-1d12a69e7c93\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbzq5" Jul 28 22:09:42.372597 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.371346 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/74555be4-d49b-4d9c-9e80-c4604c40cd23-multus-conf-dir\") pod \"multus-xvmx5\" (UID: \"74555be4-d49b-4d9c-9e80-c4604c40cd23\") " pod="openshift-multus/multus-xvmx5" Jul 28 22:09:42.372597 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.371383 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/327f1776-1215-4c6e-9fef-3e30c12b1b66-system-cni-dir\") pod \"multus-additional-cni-plugins-zcxg2\" (UID: \"327f1776-1215-4c6e-9fef-3e30c12b1b66\") " pod="openshift-multus/multus-additional-cni-plugins-zcxg2" Jul 28 22:09:42.372597 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.371383 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/74555be4-d49b-4d9c-9e80-c4604c40cd23-multus-conf-dir\") pod \"multus-xvmx5\" (UID: \"74555be4-d49b-4d9c-9e80-c4604c40cd23\") " pod="openshift-multus/multus-xvmx5" Jul 28 22:09:42.372597 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.371413 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/54dc150c-a1f7-4df4-9adf-03077bf59073-host\") pod \"node-ca-bs64q\" (UID: \"54dc150c-a1f7-4df4-9adf-03077bf59073\") " pod="openshift-image-registry/node-ca-bs64q" Jul 28 22:09:42.372597 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.371441 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/842e457b-ef75-43bc-a182-e3abca66995a-node-log\") pod \"ovnkube-node-vzbxk\" (UID: \"842e457b-ef75-43bc-a182-e3abca66995a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" Jul 28 22:09:42.372597 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.371448 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/842e457b-ef75-43bc-a182-e3abca66995a-systemd-units\") pod \"ovnkube-node-vzbxk\" (UID: \"842e457b-ef75-43bc-a182-e3abca66995a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" Jul 28 22:09:42.372597 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.371466 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/935749df-a4bd-4733-9085-1f8d1e4aa86a-etc-sysctl-conf\") pod \"tuned-6k2jz\" (UID: \"935749df-a4bd-4733-9085-1f8d1e4aa86a\") " pod="openshift-cluster-node-tuning-operator/tuned-6k2jz" Jul 28 22:09:42.372597 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.371480 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/842e457b-ef75-43bc-a182-e3abca66995a-env-overrides\") pod \"ovnkube-node-vzbxk\" (UID: \"842e457b-ef75-43bc-a182-e3abca66995a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" Jul 28 22:09:42.372597 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.371508 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/74555be4-d49b-4d9c-9e80-c4604c40cd23-host-var-lib-cni-bin\") pod \"multus-xvmx5\" (UID: \"74555be4-d49b-4d9c-9e80-c4604c40cd23\") " pod="openshift-multus/multus-xvmx5" Jul 28 22:09:42.372597 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.371535 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/74555be4-d49b-4d9c-9e80-c4604c40cd23-multus-daemon-config\") pod \"multus-xvmx5\" (UID: \"74555be4-d49b-4d9c-9e80-c4604c40cd23\") " pod="openshift-multus/multus-xvmx5" Jul 28 22:09:42.372597 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.371561 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/327f1776-1215-4c6e-9fef-3e30c12b1b66-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zcxg2\" (UID: \"327f1776-1215-4c6e-9fef-3e30c12b1b66\") " pod="openshift-multus/multus-additional-cni-plugins-zcxg2" Jul 28 22:09:42.372597 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.371561 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/935749df-a4bd-4733-9085-1f8d1e4aa86a-lib-modules\") pod \"tuned-6k2jz\" (UID: \"935749df-a4bd-4733-9085-1f8d1e4aa86a\") " pod="openshift-cluster-node-tuning-operator/tuned-6k2jz" Jul 28 22:09:42.372597 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.371598 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/327f1776-1215-4c6e-9fef-3e30c12b1b66-system-cni-dir\") pod \"multus-additional-cni-plugins-zcxg2\" (UID: \"327f1776-1215-4c6e-9fef-3e30c12b1b66\") " pod="openshift-multus/multus-additional-cni-plugins-zcxg2" Jul 28 22:09:42.372597 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.371645 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/842e457b-ef75-43bc-a182-e3abca66995a-host-cni-bin\") pod \"ovnkube-node-vzbxk\" (UID: \"842e457b-ef75-43bc-a182-e3abca66995a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" Jul 28 22:09:42.372597 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.371636 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a55216ee-cd69-4f12-af05-1d12a69e7c93-device-dir\") pod \"aws-ebs-csi-driver-node-fbzq5\" (UID: \"a55216ee-cd69-4f12-af05-1d12a69e7c93\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbzq5" Jul 28 22:09:42.372597 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.371647 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/327f1776-1215-4c6e-9fef-3e30c12b1b66-cnibin\") pod \"multus-additional-cni-plugins-zcxg2\" (UID: \"327f1776-1215-4c6e-9fef-3e30c12b1b66\") " pod="openshift-multus/multus-additional-cni-plugins-zcxg2" Jul 28 22:09:42.372597 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.371717 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/54dc150c-a1f7-4df4-9adf-03077bf59073-host\") pod \"node-ca-bs64q\" (UID: \"54dc150c-a1f7-4df4-9adf-03077bf59073\") " pod="openshift-image-registry/node-ca-bs64q" Jul 28 22:09:42.372597 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.371723 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/74555be4-d49b-4d9c-9e80-c4604c40cd23-host-run-k8s-cni-cncf-io\") pod \"multus-xvmx5\" (UID: \"74555be4-d49b-4d9c-9e80-c4604c40cd23\") " pod="openshift-multus/multus-xvmx5" Jul 28 22:09:42.373539 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.371589 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a55216ee-cd69-4f12-af05-1d12a69e7c93-device-dir\") pod \"aws-ebs-csi-driver-node-fbzq5\" (UID: \"a55216ee-cd69-4f12-af05-1d12a69e7c93\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbzq5" Jul 28 22:09:42.373539 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.371757 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/842e457b-ef75-43bc-a182-e3abca66995a-host-cni-netd\") pod \"ovnkube-node-vzbxk\" (UID: \"842e457b-ef75-43bc-a182-e3abca66995a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" Jul 28 22:09:42.373539 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.371767 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twbjv\" (UniqueName: \"kubernetes.io/projected/15e94249-e108-4426-b6f4-baf221c92a71-kube-api-access-twbjv\") pod \"network-metrics-daemon-wl29s\" (UID: \"15e94249-e108-4426-b6f4-baf221c92a71\") " pod="openshift-multus/network-metrics-daemon-wl29s" Jul 28 22:09:42.373539 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.371793 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/842e457b-ef75-43bc-a182-e3abca66995a-node-log\") pod \"ovnkube-node-vzbxk\" (UID: \"842e457b-ef75-43bc-a182-e3abca66995a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" Jul 28 22:09:42.373539 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.371797 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/842e457b-ef75-43bc-a182-e3abca66995a-run-ovn\") pod \"ovnkube-node-vzbxk\" (UID: \"842e457b-ef75-43bc-a182-e3abca66995a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" Jul 28 22:09:42.373539 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.371830 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/842e457b-ef75-43bc-a182-e3abca66995a-host-run-ovn-kubernetes\") pod \"ovnkube-node-vzbxk\" (UID: \"842e457b-ef75-43bc-a182-e3abca66995a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" Jul 28 22:09:42.373539 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.371875 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/59a1f798-88a8-4236-82f2-0a675d85920d-kubelet-config\") pod \"global-pull-secret-syncer-djzl8\" (UID: \"59a1f798-88a8-4236-82f2-0a675d85920d\") " pod="kube-system/global-pull-secret-syncer-djzl8" Jul 28 22:09:42.373539 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.371906 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0f57ff96-5ed8-4a73-9dfd-7e5d45bd163c-hosts-file\") pod \"node-resolver-w9bnn\" (UID: \"0f57ff96-5ed8-4a73-9dfd-7e5d45bd163c\") " pod="openshift-dns/node-resolver-w9bnn" Jul 28 22:09:42.373539 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.371913 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/935749df-a4bd-4733-9085-1f8d1e4aa86a-etc-sysctl-conf\") pod \"tuned-6k2jz\" (UID: \"935749df-a4bd-4733-9085-1f8d1e4aa86a\") " pod="openshift-cluster-node-tuning-operator/tuned-6k2jz" Jul 28 22:09:42.373539 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.371922 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/59a1f798-88a8-4236-82f2-0a675d85920d-dbus\") pod \"global-pull-secret-syncer-djzl8\" (UID: \"59a1f798-88a8-4236-82f2-0a675d85920d\") " pod="kube-system/global-pull-secret-syncer-djzl8" Jul 28 22:09:42.373539 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.371954 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/74555be4-d49b-4d9c-9e80-c4604c40cd23-host-var-lib-cni-bin\") pod \"multus-xvmx5\" (UID: \"74555be4-d49b-4d9c-9e80-c4604c40cd23\") " pod="openshift-multus/multus-xvmx5" Jul 28 22:09:42.373539 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.371956 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/935749df-a4bd-4733-9085-1f8d1e4aa86a-host\") pod \"tuned-6k2jz\" (UID: \"935749df-a4bd-4733-9085-1f8d1e4aa86a\") " pod="openshift-cluster-node-tuning-operator/tuned-6k2jz" Jul 28 22:09:42.373539 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.371982 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/74555be4-d49b-4d9c-9e80-c4604c40cd23-os-release\") pod \"multus-xvmx5\" (UID: \"74555be4-d49b-4d9c-9e80-c4604c40cd23\") " pod="openshift-multus/multus-xvmx5" Jul 28 22:09:42.373539 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.371998 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/74555be4-d49b-4d9c-9e80-c4604c40cd23-host-run-netns\") pod \"multus-xvmx5\" (UID: \"74555be4-d49b-4d9c-9e80-c4604c40cd23\") " pod="openshift-multus/multus-xvmx5" Jul 28 22:09:42.373539 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.371994 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/935749df-a4bd-4733-9085-1f8d1e4aa86a-host\") pod \"tuned-6k2jz\" (UID: \"935749df-a4bd-4733-9085-1f8d1e4aa86a\") " pod="openshift-cluster-node-tuning-operator/tuned-6k2jz" Jul 28 22:09:42.373539 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.372028 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cb4e193d-e952-43fb-9161-f45d39dd37ec-iptables-alerter-script\") pod \"iptables-alerter-flt69\" (UID: \"cb4e193d-e952-43fb-9161-f45d39dd37ec\") " pod="openshift-network-operator/iptables-alerter-flt69" Jul 28 22:09:42.373539 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.372035 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/842e457b-ef75-43bc-a182-e3abca66995a-run-ovn\") pod \"ovnkube-node-vzbxk\" (UID: \"842e457b-ef75-43bc-a182-e3abca66995a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" Jul 28 22:09:42.373539 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.372053 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/842e457b-ef75-43bc-a182-e3abca66995a-host-run-ovn-kubernetes\") pod \"ovnkube-node-vzbxk\" (UID: \"842e457b-ef75-43bc-a182-e3abca66995a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" Jul 28 22:09:42.374409 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.372055 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/842e457b-ef75-43bc-a182-e3abca66995a-host-kubelet\") pod \"ovnkube-node-vzbxk\" (UID: \"842e457b-ef75-43bc-a182-e3abca66995a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" Jul 28 22:09:42.374409 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.372084 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/842e457b-ef75-43bc-a182-e3abca66995a-host-kubelet\") pod \"ovnkube-node-vzbxk\" (UID: \"842e457b-ef75-43bc-a182-e3abca66995a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" Jul 28 22:09:42.374409 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.372090 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/842e457b-ef75-43bc-a182-e3abca66995a-ovnkube-script-lib\") pod \"ovnkube-node-vzbxk\" (UID: \"842e457b-ef75-43bc-a182-e3abca66995a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" Jul 28 22:09:42.374409 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.372084 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/59a1f798-88a8-4236-82f2-0a675d85920d-kubelet-config\") pod \"global-pull-secret-syncer-djzl8\" (UID: \"59a1f798-88a8-4236-82f2-0a675d85920d\") " pod="kube-system/global-pull-secret-syncer-djzl8" Jul 28 22:09:42.374409 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.372098 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/842e457b-ef75-43bc-a182-e3abca66995a-log-socket\") pod \"ovnkube-node-vzbxk\" (UID: \"842e457b-ef75-43bc-a182-e3abca66995a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" Jul 28 22:09:42.374409 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.372120 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/842e457b-ef75-43bc-a182-e3abca66995a-log-socket\") pod \"ovnkube-node-vzbxk\" (UID: \"842e457b-ef75-43bc-a182-e3abca66995a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" Jul 28 22:09:42.374409 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.372126 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a55216ee-cd69-4f12-af05-1d12a69e7c93-sys-fs\") pod \"aws-ebs-csi-driver-node-fbzq5\" (UID: \"a55216ee-cd69-4f12-af05-1d12a69e7c93\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbzq5" Jul 28 22:09:42.374409 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.372131 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27hqt\" (UniqueName: \"kubernetes.io/projected/842e457b-ef75-43bc-a182-e3abca66995a-kube-api-access-27hqt\") pod \"ovnkube-node-vzbxk\" (UID: \"842e457b-ef75-43bc-a182-e3abca66995a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" Jul 28 22:09:42.374409 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.371563 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/327f1776-1215-4c6e-9fef-3e30c12b1b66-cni-binary-copy\") pod \"multus-additional-cni-plugins-zcxg2\" (UID: \"327f1776-1215-4c6e-9fef-3e30c12b1b66\") " pod="openshift-multus/multus-additional-cni-plugins-zcxg2" Jul 28 22:09:42.374409 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.372155 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/935749df-a4bd-4733-9085-1f8d1e4aa86a-etc-modprobe-d\") pod \"tuned-6k2jz\" (UID: \"935749df-a4bd-4733-9085-1f8d1e4aa86a\") " pod="openshift-cluster-node-tuning-operator/tuned-6k2jz" Jul 28 22:09:42.374409 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.372178 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/935749df-a4bd-4733-9085-1f8d1e4aa86a-etc-kubernetes\") pod \"tuned-6k2jz\" (UID: \"935749df-a4bd-4733-9085-1f8d1e4aa86a\") " pod="openshift-cluster-node-tuning-operator/tuned-6k2jz" Jul 28 22:09:42.374409 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.372183 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/74555be4-d49b-4d9c-9e80-c4604c40cd23-host-run-netns\") pod \"multus-xvmx5\" (UID: \"74555be4-d49b-4d9c-9e80-c4604c40cd23\") " pod="openshift-multus/multus-xvmx5" Jul 28 22:09:42.374409 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.372204 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/935749df-a4bd-4733-9085-1f8d1e4aa86a-etc-systemd\") pod \"tuned-6k2jz\" (UID: \"935749df-a4bd-4733-9085-1f8d1e4aa86a\") " pod="openshift-cluster-node-tuning-operator/tuned-6k2jz" Jul 28 22:09:42.374409 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.372245 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/935749df-a4bd-4733-9085-1f8d1e4aa86a-var-lib-kubelet\") pod \"tuned-6k2jz\" (UID: \"935749df-a4bd-4733-9085-1f8d1e4aa86a\") " pod="openshift-cluster-node-tuning-operator/tuned-6k2jz" Jul 28 22:09:42.374409 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.372269 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/842e457b-ef75-43bc-a182-e3abca66995a-ovn-node-metrics-cert\") pod \"ovnkube-node-vzbxk\" (UID: \"842e457b-ef75-43bc-a182-e3abca66995a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" Jul 28 22:09:42.374409 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.372294 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/74555be4-d49b-4d9c-9e80-c4604c40cd23-cni-binary-copy\") pod \"multus-xvmx5\" (UID: \"74555be4-d49b-4d9c-9e80-c4604c40cd23\") " pod="openshift-multus/multus-xvmx5" Jul 28 22:09:42.374409 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.372320 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/74555be4-d49b-4d9c-9e80-c4604c40cd23-host-var-lib-kubelet\") pod \"multus-xvmx5\" (UID: \"74555be4-d49b-4d9c-9e80-c4604c40cd23\") " pod="openshift-multus/multus-xvmx5" Jul 28 22:09:42.374409 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.372344 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/74555be4-d49b-4d9c-9e80-c4604c40cd23-hostroot\") pod \"multus-xvmx5\" (UID: \"74555be4-d49b-4d9c-9e80-c4604c40cd23\") " pod="openshift-multus/multus-xvmx5" Jul 28 22:09:42.375484 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.372367 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/74555be4-d49b-4d9c-9e80-c4604c40cd23-host-run-multus-certs\") pod \"multus-xvmx5\" (UID: \"74555be4-d49b-4d9c-9e80-c4604c40cd23\") " pod="openshift-multus/multus-xvmx5" Jul 28 22:09:42.375484 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.372398 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b678w\" (UniqueName: \"kubernetes.io/projected/327f1776-1215-4c6e-9fef-3e30c12b1b66-kube-api-access-b678w\") pod \"multus-additional-cni-plugins-zcxg2\" (UID: \"327f1776-1215-4c6e-9fef-3e30c12b1b66\") " pod="openshift-multus/multus-additional-cni-plugins-zcxg2" Jul 28 22:09:42.375484 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.372425 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/59a1f798-88a8-4236-82f2-0a675d85920d-original-pull-secret\") pod \"global-pull-secret-syncer-djzl8\" (UID: \"59a1f798-88a8-4236-82f2-0a675d85920d\") " pod="kube-system/global-pull-secret-syncer-djzl8" Jul 28 22:09:42.375484 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.372446 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a55216ee-cd69-4f12-af05-1d12a69e7c93-registration-dir\") pod \"aws-ebs-csi-driver-node-fbzq5\" (UID: \"a55216ee-cd69-4f12-af05-1d12a69e7c93\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbzq5" Jul 28 22:09:42.375484 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.372468 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/842e457b-ef75-43bc-a182-e3abca66995a-run-openvswitch\") pod \"ovnkube-node-vzbxk\" (UID: \"842e457b-ef75-43bc-a182-e3abca66995a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" Jul 28 22:09:42.375484 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.372491 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/842e457b-ef75-43bc-a182-e3abca66995a-ovnkube-config\") pod \"ovnkube-node-vzbxk\" (UID: \"842e457b-ef75-43bc-a182-e3abca66995a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" Jul 28 22:09:42.375484 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.372516 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/74555be4-d49b-4d9c-9e80-c4604c40cd23-multus-cni-dir\") pod \"multus-xvmx5\" (UID: \"74555be4-d49b-4d9c-9e80-c4604c40cd23\") " pod="openshift-multus/multus-xvmx5" Jul 28 22:09:42.375484 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.372529 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/935749df-a4bd-4733-9085-1f8d1e4aa86a-etc-kubernetes\") pod \"tuned-6k2jz\" (UID: \"935749df-a4bd-4733-9085-1f8d1e4aa86a\") " pod="openshift-cluster-node-tuning-operator/tuned-6k2jz" Jul 28 22:09:42.375484 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.372543 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/327f1776-1215-4c6e-9fef-3e30c12b1b66-os-release\") pod \"multus-additional-cni-plugins-zcxg2\" (UID: \"327f1776-1215-4c6e-9fef-3e30c12b1b66\") " pod="openshift-multus/multus-additional-cni-plugins-zcxg2" Jul 28 22:09:42.375484 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.372567 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/327f1776-1215-4c6e-9fef-3e30c12b1b66-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zcxg2\" (UID: \"327f1776-1215-4c6e-9fef-3e30c12b1b66\") " pod="openshift-multus/multus-additional-cni-plugins-zcxg2" Jul 28 22:09:42.375484 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.372572 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/935749df-a4bd-4733-9085-1f8d1e4aa86a-etc-systemd\") pod \"tuned-6k2jz\" (UID: \"935749df-a4bd-4733-9085-1f8d1e4aa86a\") " pod="openshift-cluster-node-tuning-operator/tuned-6k2jz" Jul 28 22:09:42.375484 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.372591 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/54dc150c-a1f7-4df4-9adf-03077bf59073-serviceca\") pod \"node-ca-bs64q\" (UID: \"54dc150c-a1f7-4df4-9adf-03077bf59073\") " pod="openshift-image-registry/node-ca-bs64q" Jul 28 22:09:42.375484 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.372617 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/935749df-a4bd-4733-9085-1f8d1e4aa86a-var-lib-kubelet\") pod \"tuned-6k2jz\" (UID: \"935749df-a4bd-4733-9085-1f8d1e4aa86a\") " pod="openshift-cluster-node-tuning-operator/tuned-6k2jz" Jul 28 22:09:42.375484 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.372619 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/842e457b-ef75-43bc-a182-e3abca66995a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vzbxk\" (UID: \"842e457b-ef75-43bc-a182-e3abca66995a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" Jul 28 22:09:42.375484 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.372647 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f57ff96-5ed8-4a73-9dfd-7e5d45bd163c-tmp-dir\") pod \"node-resolver-w9bnn\" (UID: \"0f57ff96-5ed8-4a73-9dfd-7e5d45bd163c\") " pod="openshift-dns/node-resolver-w9bnn" Jul 28 22:09:42.375484 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.372668 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a55216ee-cd69-4f12-af05-1d12a69e7c93-registration-dir\") pod \"aws-ebs-csi-driver-node-fbzq5\" (UID: \"a55216ee-cd69-4f12-af05-1d12a69e7c93\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbzq5" Jul 28 22:09:42.375484 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.372674 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/74555be4-d49b-4d9c-9e80-c4604c40cd23-multus-daemon-config\") pod \"multus-xvmx5\" (UID: \"74555be4-d49b-4d9c-9e80-c4604c40cd23\") " pod="openshift-multus/multus-xvmx5" Jul 28 22:09:42.376335 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.372666 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/327f1776-1215-4c6e-9fef-3e30c12b1b66-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zcxg2\" (UID: \"327f1776-1215-4c6e-9fef-3e30c12b1b66\") " pod="openshift-multus/multus-additional-cni-plugins-zcxg2" Jul 28 22:09:42.376335 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.372720 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c93df2e7-d79a-4a98-8556-a98868dfd93e-konnectivity-ca\") pod \"konnectivity-agent-7q942\" (UID: \"c93df2e7-d79a-4a98-8556-a98868dfd93e\") " pod="kube-system/konnectivity-agent-7q942" Jul 28 22:09:42.376335 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.372745 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a55216ee-cd69-4f12-af05-1d12a69e7c93-socket-dir\") pod \"aws-ebs-csi-driver-node-fbzq5\" (UID: \"a55216ee-cd69-4f12-af05-1d12a69e7c93\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbzq5" Jul 28 22:09:42.376335 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.372770 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kthfk\" (UniqueName: \"kubernetes.io/projected/a55216ee-cd69-4f12-af05-1d12a69e7c93-kube-api-access-kthfk\") pod \"aws-ebs-csi-driver-node-fbzq5\" (UID: \"a55216ee-cd69-4f12-af05-1d12a69e7c93\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbzq5" Jul 28 22:09:42.376335 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.372797 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/59a1f798-88a8-4236-82f2-0a675d85920d-dbus\") pod \"global-pull-secret-syncer-djzl8\" (UID: \"59a1f798-88a8-4236-82f2-0a675d85920d\") " pod="kube-system/global-pull-secret-syncer-djzl8" Jul 28 22:09:42.376335 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.372824 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/842e457b-ef75-43bc-a182-e3abca66995a-host-run-netns\") pod \"ovnkube-node-vzbxk\" (UID: \"842e457b-ef75-43bc-a182-e3abca66995a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" Jul 28 22:09:42.376335 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.372863 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/842e457b-ef75-43bc-a182-e3abca66995a-run-openvswitch\") pod \"ovnkube-node-vzbxk\" (UID: \"842e457b-ef75-43bc-a182-e3abca66995a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" Jul 28 22:09:42.376335 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.372892 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/935749df-a4bd-4733-9085-1f8d1e4aa86a-etc-sysctl-d\") pod \"tuned-6k2jz\" (UID: \"935749df-a4bd-4733-9085-1f8d1e4aa86a\") " pod="openshift-cluster-node-tuning-operator/tuned-6k2jz" Jul 28 22:09:42.376335 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.372919 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/74555be4-d49b-4d9c-9e80-c4604c40cd23-cnibin\") pod \"multus-xvmx5\" (UID: \"74555be4-d49b-4d9c-9e80-c4604c40cd23\") " pod="openshift-multus/multus-xvmx5" Jul 28 22:09:42.376335 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.372948 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/74555be4-d49b-4d9c-9e80-c4604c40cd23-multus-socket-dir-parent\") pod \"multus-xvmx5\" (UID: \"74555be4-d49b-4d9c-9e80-c4604c40cd23\") " pod="openshift-multus/multus-xvmx5" Jul 28 22:09:42.376335 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.372976 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5cns\" (UniqueName: \"kubernetes.io/projected/74555be4-d49b-4d9c-9e80-c4604c40cd23-kube-api-access-j5cns\") pod \"multus-xvmx5\" (UID: \"74555be4-d49b-4d9c-9e80-c4604c40cd23\") " pod="openshift-multus/multus-xvmx5" Jul 28 22:09:42.376335 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.373008 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/327f1776-1215-4c6e-9fef-3e30c12b1b66-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zcxg2\" (UID: \"327f1776-1215-4c6e-9fef-3e30c12b1b66\") " pod="openshift-multus/multus-additional-cni-plugins-zcxg2" Jul 28 22:09:42.376335 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.373186 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/327f1776-1215-4c6e-9fef-3e30c12b1b66-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zcxg2\" (UID: \"327f1776-1215-4c6e-9fef-3e30c12b1b66\") " pod="openshift-multus/multus-additional-cni-plugins-zcxg2" Jul 28 22:09:42.376335 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.373237 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/74555be4-d49b-4d9c-9e80-c4604c40cd23-cni-binary-copy\") pod \"multus-xvmx5\" (UID: \"74555be4-d49b-4d9c-9e80-c4604c40cd23\") " pod="openshift-multus/multus-xvmx5" Jul 28 22:09:42.376335 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.373292 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/74555be4-d49b-4d9c-9e80-c4604c40cd23-host-run-multus-certs\") pod \"multus-xvmx5\" (UID: \"74555be4-d49b-4d9c-9e80-c4604c40cd23\") " pod="openshift-multus/multus-xvmx5" Jul 28 22:09:42.376335 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.373301 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/327f1776-1215-4c6e-9fef-3e30c12b1b66-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zcxg2\" (UID: \"327f1776-1215-4c6e-9fef-3e30c12b1b66\") " pod="openshift-multus/multus-additional-cni-plugins-zcxg2" Jul 28 22:09:42.376335 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.373331 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/74555be4-d49b-4d9c-9e80-c4604c40cd23-host-var-lib-kubelet\") pod \"multus-xvmx5\" (UID: \"74555be4-d49b-4d9c-9e80-c4604c40cd23\") " pod="openshift-multus/multus-xvmx5" Jul 28 22:09:42.377169 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.373351 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/842e457b-ef75-43bc-a182-e3abca66995a-host-run-netns\") pod \"ovnkube-node-vzbxk\" (UID: \"842e457b-ef75-43bc-a182-e3abca66995a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" Jul 28 22:09:42.377169 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.373373 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/74555be4-d49b-4d9c-9e80-c4604c40cd23-hostroot\") pod \"multus-xvmx5\" (UID: \"74555be4-d49b-4d9c-9e80-c4604c40cd23\") " pod="openshift-multus/multus-xvmx5" Jul 28 22:09:42.377169 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.373393 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c93df2e7-d79a-4a98-8556-a98868dfd93e-konnectivity-ca\") pod \"konnectivity-agent-7q942\" (UID: \"c93df2e7-d79a-4a98-8556-a98868dfd93e\") " pod="kube-system/konnectivity-agent-7q942" Jul 28 22:09:42.377169 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:42.373457 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 22:09:42.377169 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:42.373517 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59a1f798-88a8-4236-82f2-0a675d85920d-original-pull-secret podName:59a1f798-88a8-4236-82f2-0a675d85920d nodeName:}" failed. No retries permitted until 2026-07-28 22:09:42.873497017 +0000 UTC m=+2.960755083 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/59a1f798-88a8-4236-82f2-0a675d85920d-original-pull-secret") pod "global-pull-secret-syncer-djzl8" (UID: "59a1f798-88a8-4236-82f2-0a675d85920d") : object "kube-system"/"original-pull-secret" not registered Jul 28 22:09:42.377169 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.373642 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/842e457b-ef75-43bc-a182-e3abca66995a-ovnkube-config\") pod \"ovnkube-node-vzbxk\" (UID: \"842e457b-ef75-43bc-a182-e3abca66995a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" Jul 28 22:09:42.377169 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.372490 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/935749df-a4bd-4733-9085-1f8d1e4aa86a-etc-modprobe-d\") pod \"tuned-6k2jz\" (UID: \"935749df-a4bd-4733-9085-1f8d1e4aa86a\") " pod="openshift-cluster-node-tuning-operator/tuned-6k2jz" Jul 28 22:09:42.377169 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.373675 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a55216ee-cd69-4f12-af05-1d12a69e7c93-socket-dir\") pod \"aws-ebs-csi-driver-node-fbzq5\" (UID: \"a55216ee-cd69-4f12-af05-1d12a69e7c93\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbzq5" Jul 28 22:09:42.377169 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.373756 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/327f1776-1215-4c6e-9fef-3e30c12b1b66-os-release\") pod \"multus-additional-cni-plugins-zcxg2\" (UID: \"327f1776-1215-4c6e-9fef-3e30c12b1b66\") " pod="openshift-multus/multus-additional-cni-plugins-zcxg2" Jul 28 22:09:42.377169 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.373770 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/54dc150c-a1f7-4df4-9adf-03077bf59073-serviceca\") pod \"node-ca-bs64q\" (UID: \"54dc150c-a1f7-4df4-9adf-03077bf59073\") " pod="openshift-image-registry/node-ca-bs64q" Jul 28 22:09:42.377169 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.373776 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/842e457b-ef75-43bc-a182-e3abca66995a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vzbxk\" (UID: \"842e457b-ef75-43bc-a182-e3abca66995a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" Jul 28 22:09:42.377169 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.373801 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/74555be4-d49b-4d9c-9e80-c4604c40cd23-multus-cni-dir\") pod \"multus-xvmx5\" (UID: \"74555be4-d49b-4d9c-9e80-c4604c40cd23\") " pod="openshift-multus/multus-xvmx5" Jul 28 22:09:42.377169 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.373826 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/74555be4-d49b-4d9c-9e80-c4604c40cd23-cnibin\") pod \"multus-xvmx5\" (UID: \"74555be4-d49b-4d9c-9e80-c4604c40cd23\") " pod="openshift-multus/multus-xvmx5" Jul 28 22:09:42.377169 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.373831 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/74555be4-d49b-4d9c-9e80-c4604c40cd23-multus-socket-dir-parent\") pod \"multus-xvmx5\" (UID: \"74555be4-d49b-4d9c-9e80-c4604c40cd23\") " pod="openshift-multus/multus-xvmx5" Jul 28 22:09:42.377169 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.373918 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/935749df-a4bd-4733-9085-1f8d1e4aa86a-etc-sysctl-d\") pod \"tuned-6k2jz\" (UID: \"935749df-a4bd-4733-9085-1f8d1e4aa86a\") " pod="openshift-cluster-node-tuning-operator/tuned-6k2jz" Jul 28 22:09:42.377169 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.374145 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f57ff96-5ed8-4a73-9dfd-7e5d45bd163c-tmp-dir\") pod \"node-resolver-w9bnn\" (UID: \"0f57ff96-5ed8-4a73-9dfd-7e5d45bd163c\") " pod="openshift-dns/node-resolver-w9bnn" Jul 28 22:09:42.377169 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:42.375501 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 22:09:42.377169 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:42.375519 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 22:09:42.377956 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:42.375532 2553 projected.go:196] Error preparing data for projected volume kube-api-access-8m97d for pod openshift-network-diagnostics/network-check-target-qz6gw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 22:09:42.377956 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:42.375601 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b595cc6-bee1-476f-842c-00ed98dd3e79-kube-api-access-8m97d podName:3b595cc6-bee1-476f-842c-00ed98dd3e79 nodeName:}" failed. No retries permitted until 2026-07-28 22:09:42.875585289 +0000 UTC m=+2.962843349 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-8m97d" (UniqueName: "kubernetes.io/projected/3b595cc6-bee1-476f-842c-00ed98dd3e79-kube-api-access-8m97d") pod "network-check-target-qz6gw" (UID: "3b595cc6-bee1-476f-842c-00ed98dd3e79") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 22:09:42.377956 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.375944 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/935749df-a4bd-4733-9085-1f8d1e4aa86a-etc-tuned\") pod \"tuned-6k2jz\" (UID: \"935749df-a4bd-4733-9085-1f8d1e4aa86a\") " pod="openshift-cluster-node-tuning-operator/tuned-6k2jz" Jul 28 22:09:42.377956 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.376537 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/842e457b-ef75-43bc-a182-e3abca66995a-ovn-node-metrics-cert\") pod \"ovnkube-node-vzbxk\" (UID: \"842e457b-ef75-43bc-a182-e3abca66995a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" Jul 28 22:09:42.377956 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.376961 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c93df2e7-d79a-4a98-8556-a98868dfd93e-agent-certs\") pod \"konnectivity-agent-7q942\" (UID: \"c93df2e7-d79a-4a98-8556-a98868dfd93e\") " pod="kube-system/konnectivity-agent-7q942" Jul 28 22:09:42.377956 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.377549 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/935749df-a4bd-4733-9085-1f8d1e4aa86a-tmp\") pod \"tuned-6k2jz\" (UID: \"935749df-a4bd-4733-9085-1f8d1e4aa86a\") " pod="openshift-cluster-node-tuning-operator/tuned-6k2jz" Jul 28 22:09:42.378492 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.378470 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgpqd\" (UniqueName: \"kubernetes.io/projected/0f57ff96-5ed8-4a73-9dfd-7e5d45bd163c-kube-api-access-kgpqd\") pod \"node-resolver-w9bnn\" (UID: \"0f57ff96-5ed8-4a73-9dfd-7e5d45bd163c\") " pod="openshift-dns/node-resolver-w9bnn" Jul 28 22:09:42.378734 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.378692 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8czw2\" (UniqueName: \"kubernetes.io/projected/935749df-a4bd-4733-9085-1f8d1e4aa86a-kube-api-access-8czw2\") pod \"tuned-6k2jz\" (UID: \"935749df-a4bd-4733-9085-1f8d1e4aa86a\") " pod="openshift-cluster-node-tuning-operator/tuned-6k2jz" Jul 28 22:09:42.379027 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.379010 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx25l\" (UniqueName: \"kubernetes.io/projected/54dc150c-a1f7-4df4-9adf-03077bf59073-kube-api-access-hx25l\") pod \"node-ca-bs64q\" (UID: \"54dc150c-a1f7-4df4-9adf-03077bf59073\") " pod="openshift-image-registry/node-ca-bs64q" Jul 28 22:09:42.383772 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.383748 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27hqt\" (UniqueName: \"kubernetes.io/projected/842e457b-ef75-43bc-a182-e3abca66995a-kube-api-access-27hqt\") pod \"ovnkube-node-vzbxk\" (UID: \"842e457b-ef75-43bc-a182-e3abca66995a\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" Jul 28 22:09:42.385520 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.385497 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b678w\" (UniqueName: \"kubernetes.io/projected/327f1776-1215-4c6e-9fef-3e30c12b1b66-kube-api-access-b678w\") pod \"multus-additional-cni-plugins-zcxg2\" (UID: \"327f1776-1215-4c6e-9fef-3e30c12b1b66\") " pod="openshift-multus/multus-additional-cni-plugins-zcxg2" Jul 28 22:09:42.386148 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.386126 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kthfk\" (UniqueName: \"kubernetes.io/projected/a55216ee-cd69-4f12-af05-1d12a69e7c93-kube-api-access-kthfk\") pod \"aws-ebs-csi-driver-node-fbzq5\" (UID: \"a55216ee-cd69-4f12-af05-1d12a69e7c93\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbzq5" Jul 28 22:09:42.387483 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.387460 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5cns\" (UniqueName: \"kubernetes.io/projected/74555be4-d49b-4d9c-9e80-c4604c40cd23-kube-api-access-j5cns\") pod \"multus-xvmx5\" (UID: \"74555be4-d49b-4d9c-9e80-c4604c40cd23\") " pod="openshift-multus/multus-xvmx5" Jul 28 22:09:42.418168 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.418042 2553 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 22:09:42.473655 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.473620 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twbjv\" (UniqueName: \"kubernetes.io/projected/15e94249-e108-4426-b6f4-baf221c92a71-kube-api-access-twbjv\") pod \"network-metrics-daemon-wl29s\" (UID: \"15e94249-e108-4426-b6f4-baf221c92a71\") " pod="openshift-multus/network-metrics-daemon-wl29s" Jul 28 22:09:42.473863 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.473670 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cb4e193d-e952-43fb-9161-f45d39dd37ec-iptables-alerter-script\") pod \"iptables-alerter-flt69\" (UID: \"cb4e193d-e952-43fb-9161-f45d39dd37ec\") " pod="openshift-network-operator/iptables-alerter-flt69" Jul 28 22:09:42.474043 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.474017 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwkwh\" (UniqueName: \"kubernetes.io/projected/cb4e193d-e952-43fb-9161-f45d39dd37ec-kube-api-access-wwkwh\") pod \"iptables-alerter-flt69\" (UID: \"cb4e193d-e952-43fb-9161-f45d39dd37ec\") " pod="openshift-network-operator/iptables-alerter-flt69" Jul 28 22:09:42.474115 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.474076 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15e94249-e108-4426-b6f4-baf221c92a71-metrics-certs\") pod \"network-metrics-daemon-wl29s\" (UID: \"15e94249-e108-4426-b6f4-baf221c92a71\") " pod="openshift-multus/network-metrics-daemon-wl29s" Jul 28 22:09:42.474115 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.474101 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cb4e193d-e952-43fb-9161-f45d39dd37ec-host-slash\") pod \"iptables-alerter-flt69\" (UID: \"cb4e193d-e952-43fb-9161-f45d39dd37ec\") " pod="openshift-network-operator/iptables-alerter-flt69" Jul 28 22:09:42.474208 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.474175 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cb4e193d-e952-43fb-9161-f45d39dd37ec-host-slash\") pod \"iptables-alerter-flt69\" (UID: \"cb4e193d-e952-43fb-9161-f45d39dd37ec\") " pod="openshift-network-operator/iptables-alerter-flt69" Jul 28 22:09:42.474280 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.474260 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cb4e193d-e952-43fb-9161-f45d39dd37ec-iptables-alerter-script\") pod \"iptables-alerter-flt69\" (UID: \"cb4e193d-e952-43fb-9161-f45d39dd37ec\") " pod="openshift-network-operator/iptables-alerter-flt69" Jul 28 22:09:42.474381 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:42.474365 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 22:09:42.474439 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:42.474434 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15e94249-e108-4426-b6f4-baf221c92a71-metrics-certs podName:15e94249-e108-4426-b6f4-baf221c92a71 nodeName:}" failed. No retries permitted until 2026-07-28 22:09:42.974419175 +0000 UTC m=+3.061677222 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/15e94249-e108-4426-b6f4-baf221c92a71-metrics-certs") pod "network-metrics-daemon-wl29s" (UID: "15e94249-e108-4426-b6f4-baf221c92a71") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 22:09:42.482533 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.482508 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twbjv\" (UniqueName: \"kubernetes.io/projected/15e94249-e108-4426-b6f4-baf221c92a71-kube-api-access-twbjv\") pod \"network-metrics-daemon-wl29s\" (UID: \"15e94249-e108-4426-b6f4-baf221c92a71\") " pod="openshift-multus/network-metrics-daemon-wl29s" Jul 28 22:09:42.482654 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.482508 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwkwh\" (UniqueName: \"kubernetes.io/projected/cb4e193d-e952-43fb-9161-f45d39dd37ec-kube-api-access-wwkwh\") pod \"iptables-alerter-flt69\" (UID: \"cb4e193d-e952-43fb-9161-f45d39dd37ec\") " pod="openshift-network-operator/iptables-alerter-flt69" Jul 28 22:09:42.561639 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.561549 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" Jul 28 22:09:42.568731 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.568687 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6k2jz" Jul 28 22:09:42.580768 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.580736 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-w9bnn" Jul 28 22:09:42.586854 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.586831 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bs64q" Jul 28 22:09:42.594592 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.594572 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xvmx5" Jul 28 22:09:42.602211 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.602190 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7q942" Jul 28 22:09:42.610127 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.610104 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbzq5" Jul 28 22:09:42.617686 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.617657 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zcxg2" Jul 28 22:09:42.624375 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.624350 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-flt69" Jul 28 22:09:42.876958 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.876857 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/59a1f798-88a8-4236-82f2-0a675d85920d-original-pull-secret\") pod \"global-pull-secret-syncer-djzl8\" (UID: \"59a1f798-88a8-4236-82f2-0a675d85920d\") " pod="kube-system/global-pull-secret-syncer-djzl8" Jul 28 22:09:42.876958 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.876928 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m97d\" (UniqueName: \"kubernetes.io/projected/3b595cc6-bee1-476f-842c-00ed98dd3e79-kube-api-access-8m97d\") pod \"network-check-target-qz6gw\" (UID: \"3b595cc6-bee1-476f-842c-00ed98dd3e79\") " pod="openshift-network-diagnostics/network-check-target-qz6gw" Jul 28 22:09:42.877166 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:42.877016 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 22:09:42.877166 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:42.877065 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 22:09:42.877166 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:42.877083 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 22:09:42.877166 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:42.877094 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59a1f798-88a8-4236-82f2-0a675d85920d-original-pull-secret podName:59a1f798-88a8-4236-82f2-0a675d85920d nodeName:}" failed. No retries permitted until 2026-07-28 22:09:43.877072841 +0000 UTC m=+3.964330911 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/59a1f798-88a8-4236-82f2-0a675d85920d-original-pull-secret") pod "global-pull-secret-syncer-djzl8" (UID: "59a1f798-88a8-4236-82f2-0a675d85920d") : object "kube-system"/"original-pull-secret" not registered Jul 28 22:09:42.877166 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:42.877098 2553 projected.go:196] Error preparing data for projected volume kube-api-access-8m97d for pod openshift-network-diagnostics/network-check-target-qz6gw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 22:09:42.877166 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:42.877150 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b595cc6-bee1-476f-842c-00ed98dd3e79-kube-api-access-8m97d podName:3b595cc6-bee1-476f-842c-00ed98dd3e79 nodeName:}" failed. No retries permitted until 2026-07-28 22:09:43.877133996 +0000 UTC m=+3.964392061 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-8m97d" (UniqueName: "kubernetes.io/projected/3b595cc6-bee1-476f-842c-00ed98dd3e79-kube-api-access-8m97d") pod "network-check-target-qz6gw" (UID: "3b595cc6-bee1-476f-842c-00ed98dd3e79") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 22:09:42.977527 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:42.977493 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15e94249-e108-4426-b6f4-baf221c92a71-metrics-certs\") pod \"network-metrics-daemon-wl29s\" (UID: \"15e94249-e108-4426-b6f4-baf221c92a71\") " pod="openshift-multus/network-metrics-daemon-wl29s" Jul 28 22:09:42.977687 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:42.977631 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 22:09:42.977766 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:42.977754 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15e94249-e108-4426-b6f4-baf221c92a71-metrics-certs podName:15e94249-e108-4426-b6f4-baf221c92a71 nodeName:}" failed. No retries permitted until 2026-07-28 22:09:43.977683296 +0000 UTC m=+4.064941343 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/15e94249-e108-4426-b6f4-baf221c92a71-metrics-certs") pod "network-metrics-daemon-wl29s" (UID: "15e94249-e108-4426-b6f4-baf221c92a71") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 22:09:43.296986 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:43.296953 2553 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 22:04:42 +0000 UTC" deadline="2027-12-27 04:21:01.062454248 +0000 UTC" Jul 28 22:09:43.296986 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:43.296983 2553 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12390h11m17.765474936s" Jul 28 22:09:43.303562 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:43.303537 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f57ff96_5ed8_4a73_9dfd_7e5d45bd163c.slice/crio-0fe520e46497e278e9d584ed77537a285132327352bf36228fecf1845c8ca502 WatchSource:0}: Error finding container 0fe520e46497e278e9d584ed77537a285132327352bf36228fecf1845c8ca502: Status 404 returned error can't find the container with id 0fe520e46497e278e9d584ed77537a285132327352bf36228fecf1845c8ca502 Jul 28 22:09:43.305005 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:43.304968 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54dc150c_a1f7_4df4_9adf_03077bf59073.slice/crio-7de0900bc61444d2c24a89af6e1a06aba3c6c1fc810482c6f8c15f73329b3c95 WatchSource:0}: Error finding container 7de0900bc61444d2c24a89af6e1a06aba3c6c1fc810482c6f8c15f73329b3c95: Status 404 returned error can't find the container with id 7de0900bc61444d2c24a89af6e1a06aba3c6c1fc810482c6f8c15f73329b3c95 Jul 28 22:09:43.324818 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:43.324724 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod935749df_a4bd_4733_9085_1f8d1e4aa86a.slice/crio-f76509f3f95e422992e47cb52a9099ae117dc7682865c9e85b032ba6445535d3 WatchSource:0}: Error finding container f76509f3f95e422992e47cb52a9099ae117dc7682865c9e85b032ba6445535d3: Status 404 returned error can't find the container with id f76509f3f95e422992e47cb52a9099ae117dc7682865c9e85b032ba6445535d3 Jul 28 22:09:43.327671 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:43.327647 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda55216ee_cd69_4f12_af05_1d12a69e7c93.slice/crio-c06762eea966d6504706a7449672d43b126858b47958249377968e09fbab1dec WatchSource:0}: Error finding container c06762eea966d6504706a7449672d43b126858b47958249377968e09fbab1dec: Status 404 returned error can't find the container with id c06762eea966d6504706a7449672d43b126858b47958249377968e09fbab1dec Jul 28 22:09:43.328869 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:43.328842 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc93df2e7_d79a_4a98_8556_a98868dfd93e.slice/crio-916dcd3f2f3e29738aff9ed54058ae6f0015e3d384a3c969346b79343f7a4014 WatchSource:0}: Error finding container 916dcd3f2f3e29738aff9ed54058ae6f0015e3d384a3c969346b79343f7a4014: Status 404 returned error can't find the container with id 916dcd3f2f3e29738aff9ed54058ae6f0015e3d384a3c969346b79343f7a4014 Jul 28 22:09:43.330299 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:43.329898 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74555be4_d49b_4d9c_9e80_c4604c40cd23.slice/crio-92506e999ce1d52551aef9af65c21c050d0ddba9cecaa376f48f2f1a3c814c38 WatchSource:0}: Error finding container 92506e999ce1d52551aef9af65c21c050d0ddba9cecaa376f48f2f1a3c814c38: Status 404 returned error can't find the container with id 92506e999ce1d52551aef9af65c21c050d0ddba9cecaa376f48f2f1a3c814c38 Jul 28 22:09:43.330917 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:43.330810 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod327f1776_1215_4c6e_9fef_3e30c12b1b66.slice/crio-120e45442f61fc30916a9a0b7ae483094a018b7dc5b9c3071fabfc0f3f70b6ea WatchSource:0}: Error finding container 120e45442f61fc30916a9a0b7ae483094a018b7dc5b9c3071fabfc0f3f70b6ea: Status 404 returned error can't find the container with id 120e45442f61fc30916a9a0b7ae483094a018b7dc5b9c3071fabfc0f3f70b6ea Jul 28 22:09:43.332316 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:43.332284 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod842e457b_ef75_43bc_a182_e3abca66995a.slice/crio-dceda361ed379ea20ad15eb5808eae71a4f4e42c2b93f38e13cd6c1490e5bbe3 WatchSource:0}: Error finding container dceda361ed379ea20ad15eb5808eae71a4f4e42c2b93f38e13cd6c1490e5bbe3: Status 404 returned error can't find the container with id dceda361ed379ea20ad15eb5808eae71a4f4e42c2b93f38e13cd6c1490e5bbe3 Jul 28 22:09:43.335016 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:09:43.334997 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb4e193d_e952_43fb_9161_f45d39dd37ec.slice/crio-a61c4309a17d16eb8e7cf167da14ad508d7d99107f4fd3f5fb4ad2d78bdaca57 WatchSource:0}: Error finding container a61c4309a17d16eb8e7cf167da14ad508d7d99107f4fd3f5fb4ad2d78bdaca57: Status 404 returned error can't find the container with id a61c4309a17d16eb8e7cf167da14ad508d7d99107f4fd3f5fb4ad2d78bdaca57 Jul 28 22:09:43.399538 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:43.399396 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-djzl8" Jul 28 22:09:43.399649 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:43.399616 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-djzl8" podUID="59a1f798-88a8-4236-82f2-0a675d85920d" Jul 28 22:09:43.410045 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:43.410004 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" event={"ID":"842e457b-ef75-43bc-a182-e3abca66995a","Type":"ContainerStarted","Data":"dceda361ed379ea20ad15eb5808eae71a4f4e42c2b93f38e13cd6c1490e5bbe3"} Jul 28 22:09:43.412178 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:43.412147 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xvmx5" event={"ID":"74555be4-d49b-4d9c-9e80-c4604c40cd23","Type":"ContainerStarted","Data":"92506e999ce1d52551aef9af65c21c050d0ddba9cecaa376f48f2f1a3c814c38"} Jul 28 22:09:43.413240 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:43.413213 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6k2jz" event={"ID":"935749df-a4bd-4733-9085-1f8d1e4aa86a","Type":"ContainerStarted","Data":"f76509f3f95e422992e47cb52a9099ae117dc7682865c9e85b032ba6445535d3"} Jul 28 22:09:43.414619 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:43.414597 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bs64q" event={"ID":"54dc150c-a1f7-4df4-9adf-03077bf59073","Type":"ContainerStarted","Data":"7de0900bc61444d2c24a89af6e1a06aba3c6c1fc810482c6f8c15f73329b3c95"} Jul 28 22:09:43.417548 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:43.417515 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-flt69" event={"ID":"cb4e193d-e952-43fb-9161-f45d39dd37ec","Type":"ContainerStarted","Data":"a61c4309a17d16eb8e7cf167da14ad508d7d99107f4fd3f5fb4ad2d78bdaca57"} Jul 28 22:09:43.418640 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:43.418604 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zcxg2" event={"ID":"327f1776-1215-4c6e-9fef-3e30c12b1b66","Type":"ContainerStarted","Data":"120e45442f61fc30916a9a0b7ae483094a018b7dc5b9c3071fabfc0f3f70b6ea"} Jul 28 22:09:43.419777 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:43.419732 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7q942" event={"ID":"c93df2e7-d79a-4a98-8556-a98868dfd93e","Type":"ContainerStarted","Data":"916dcd3f2f3e29738aff9ed54058ae6f0015e3d384a3c969346b79343f7a4014"} Jul 28 22:09:43.420869 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:43.420846 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbzq5" event={"ID":"a55216ee-cd69-4f12-af05-1d12a69e7c93","Type":"ContainerStarted","Data":"c06762eea966d6504706a7449672d43b126858b47958249377968e09fbab1dec"} Jul 28 22:09:43.421790 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:43.421765 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-w9bnn" event={"ID":"0f57ff96-5ed8-4a73-9dfd-7e5d45bd163c","Type":"ContainerStarted","Data":"0fe520e46497e278e9d584ed77537a285132327352bf36228fecf1845c8ca502"} Jul 28 22:09:43.430244 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:43.430204 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-132.ec2.internal" podStartSLOduration=2.430189933 podStartE2EDuration="2.430189933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 22:09:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:09:43.430187635 +0000 UTC m=+3.517445705" watchObservedRunningTime="2026-07-28 22:09:43.430189933 +0000 UTC m=+3.517448002" Jul 28 22:09:43.478518 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:43.478483 2553 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 22:09:43.886199 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:43.885763 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/59a1f798-88a8-4236-82f2-0a675d85920d-original-pull-secret\") pod \"global-pull-secret-syncer-djzl8\" (UID: \"59a1f798-88a8-4236-82f2-0a675d85920d\") " pod="kube-system/global-pull-secret-syncer-djzl8" Jul 28 22:09:43.886199 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:43.885846 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m97d\" (UniqueName: \"kubernetes.io/projected/3b595cc6-bee1-476f-842c-00ed98dd3e79-kube-api-access-8m97d\") pod \"network-check-target-qz6gw\" (UID: \"3b595cc6-bee1-476f-842c-00ed98dd3e79\") " pod="openshift-network-diagnostics/network-check-target-qz6gw" Jul 28 22:09:43.886199 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:43.885917 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 22:09:43.886199 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:43.885991 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59a1f798-88a8-4236-82f2-0a675d85920d-original-pull-secret podName:59a1f798-88a8-4236-82f2-0a675d85920d nodeName:}" failed. No retries permitted until 2026-07-28 22:09:45.885973444 +0000 UTC m=+5.973231507 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/59a1f798-88a8-4236-82f2-0a675d85920d-original-pull-secret") pod "global-pull-secret-syncer-djzl8" (UID: "59a1f798-88a8-4236-82f2-0a675d85920d") : object "kube-system"/"original-pull-secret" not registered Jul 28 22:09:43.886199 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:43.885991 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 22:09:43.886199 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:43.886014 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 22:09:43.886199 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:43.886027 2553 projected.go:196] Error preparing data for projected volume kube-api-access-8m97d for pod openshift-network-diagnostics/network-check-target-qz6gw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 22:09:43.886199 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:43.886065 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b595cc6-bee1-476f-842c-00ed98dd3e79-kube-api-access-8m97d podName:3b595cc6-bee1-476f-842c-00ed98dd3e79 nodeName:}" failed. No retries permitted until 2026-07-28 22:09:45.886053725 +0000 UTC m=+5.973311785 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-8m97d" (UniqueName: "kubernetes.io/projected/3b595cc6-bee1-476f-842c-00ed98dd3e79-kube-api-access-8m97d") pod "network-check-target-qz6gw" (UID: "3b595cc6-bee1-476f-842c-00ed98dd3e79") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 22:09:43.986880 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:43.986842 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15e94249-e108-4426-b6f4-baf221c92a71-metrics-certs\") pod \"network-metrics-daemon-wl29s\" (UID: \"15e94249-e108-4426-b6f4-baf221c92a71\") " pod="openshift-multus/network-metrics-daemon-wl29s" Jul 28 22:09:43.987116 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:43.987097 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 22:09:43.987184 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:43.987172 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15e94249-e108-4426-b6f4-baf221c92a71-metrics-certs podName:15e94249-e108-4426-b6f4-baf221c92a71 nodeName:}" failed. No retries permitted until 2026-07-28 22:09:45.987152467 +0000 UTC m=+6.074410528 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/15e94249-e108-4426-b6f4-baf221c92a71-metrics-certs") pod "network-metrics-daemon-wl29s" (UID: "15e94249-e108-4426-b6f4-baf221c92a71") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 22:09:44.301885 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:44.297648 2553 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 22:04:42 +0000 UTC" deadline="2028-03-16 21:25:41.656013955 +0000 UTC" Jul 28 22:09:44.301885 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:44.297721 2553 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14327h15m57.358297174s" Jul 28 22:09:44.400303 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:44.399826 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wl29s" Jul 28 22:09:44.400303 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:44.399965 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wl29s" podUID="15e94249-e108-4426-b6f4-baf221c92a71" Jul 28 22:09:44.400748 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:44.400598 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qz6gw" Jul 28 22:09:44.400748 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:44.400695 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qz6gw" podUID="3b595cc6-bee1-476f-842c-00ed98dd3e79" Jul 28 22:09:44.435453 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:44.435419 2553 generic.go:356] "Generic (PLEG): container finished" podID="a56b594b11650f501e0866607be01ff3" containerID="e5c7fd45e3a2a6beee2a606097baf4fab5ac22a32df9e8af20a702153a5846c6" exitCode=0 Jul 28 22:09:44.435619 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:44.435523 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-132.ec2.internal" event={"ID":"a56b594b11650f501e0866607be01ff3","Type":"ContainerDied","Data":"e5c7fd45e3a2a6beee2a606097baf4fab5ac22a32df9e8af20a702153a5846c6"} Jul 28 22:09:44.445339 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:44.445311 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-132.ec2.internal" event={"ID":"84e2d03b22c44b29314bad0b0a3a050e","Type":"ContainerStarted","Data":"e8abb42556cb1e8ce8dbe35a426117236e67c8ed96463f17283632ce549164cd"} Jul 28 22:09:45.399171 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:45.399139 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-djzl8" Jul 28 22:09:45.399689 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:45.399283 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-djzl8" podUID="59a1f798-88a8-4236-82f2-0a675d85920d" Jul 28 22:09:45.459201 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:45.459163 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-132.ec2.internal" event={"ID":"a56b594b11650f501e0866607be01ff3","Type":"ContainerStarted","Data":"3be08ce24e925846d4c5c8f44deac0115eb27641cd7b165a040eab75c3629cee"} Jul 28 22:09:45.906127 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:45.905928 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m97d\" (UniqueName: \"kubernetes.io/projected/3b595cc6-bee1-476f-842c-00ed98dd3e79-kube-api-access-8m97d\") pod \"network-check-target-qz6gw\" (UID: \"3b595cc6-bee1-476f-842c-00ed98dd3e79\") " pod="openshift-network-diagnostics/network-check-target-qz6gw" Jul 28 22:09:45.906127 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:45.906026 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/59a1f798-88a8-4236-82f2-0a675d85920d-original-pull-secret\") pod \"global-pull-secret-syncer-djzl8\" (UID: \"59a1f798-88a8-4236-82f2-0a675d85920d\") " pod="kube-system/global-pull-secret-syncer-djzl8" Jul 28 22:09:45.906435 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:45.906151 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 22:09:45.906435 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:45.906216 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59a1f798-88a8-4236-82f2-0a675d85920d-original-pull-secret podName:59a1f798-88a8-4236-82f2-0a675d85920d nodeName:}" failed. No retries permitted until 2026-07-28 22:09:49.906196583 +0000 UTC m=+9.993454652 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/59a1f798-88a8-4236-82f2-0a675d85920d-original-pull-secret") pod "global-pull-secret-syncer-djzl8" (UID: "59a1f798-88a8-4236-82f2-0a675d85920d") : object "kube-system"/"original-pull-secret" not registered Jul 28 22:09:45.906643 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:45.906612 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 22:09:45.906643 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:45.906633 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 22:09:45.906836 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:45.906646 2553 projected.go:196] Error preparing data for projected volume kube-api-access-8m97d for pod openshift-network-diagnostics/network-check-target-qz6gw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 22:09:45.906836 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:45.906692 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b595cc6-bee1-476f-842c-00ed98dd3e79-kube-api-access-8m97d podName:3b595cc6-bee1-476f-842c-00ed98dd3e79 nodeName:}" failed. No retries permitted until 2026-07-28 22:09:49.906675969 +0000 UTC m=+9.993934023 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-8m97d" (UniqueName: "kubernetes.io/projected/3b595cc6-bee1-476f-842c-00ed98dd3e79-kube-api-access-8m97d") pod "network-check-target-qz6gw" (UID: "3b595cc6-bee1-476f-842c-00ed98dd3e79") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 22:09:46.007311 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:46.006612 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15e94249-e108-4426-b6f4-baf221c92a71-metrics-certs\") pod \"network-metrics-daemon-wl29s\" (UID: \"15e94249-e108-4426-b6f4-baf221c92a71\") " pod="openshift-multus/network-metrics-daemon-wl29s" Jul 28 22:09:46.007311 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:46.006862 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 22:09:46.007311 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:46.006931 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15e94249-e108-4426-b6f4-baf221c92a71-metrics-certs podName:15e94249-e108-4426-b6f4-baf221c92a71 nodeName:}" failed. No retries permitted until 2026-07-28 22:09:50.006908767 +0000 UTC m=+10.094166844 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/15e94249-e108-4426-b6f4-baf221c92a71-metrics-certs") pod "network-metrics-daemon-wl29s" (UID: "15e94249-e108-4426-b6f4-baf221c92a71") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 22:09:46.399792 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:46.399203 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wl29s" Jul 28 22:09:46.399792 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:46.399428 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wl29s" podUID="15e94249-e108-4426-b6f4-baf221c92a71" Jul 28 22:09:46.400610 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:46.400408 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qz6gw" Jul 28 22:09:46.400610 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:46.400525 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qz6gw" podUID="3b595cc6-bee1-476f-842c-00ed98dd3e79" Jul 28 22:09:47.399430 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:47.399395 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-djzl8" Jul 28 22:09:47.399632 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:47.399531 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-djzl8" podUID="59a1f798-88a8-4236-82f2-0a675d85920d" Jul 28 22:09:48.399872 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:48.399831 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wl29s" Jul 28 22:09:48.400296 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:48.399981 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wl29s" podUID="15e94249-e108-4426-b6f4-baf221c92a71" Jul 28 22:09:48.400296 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:48.400048 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qz6gw" Jul 28 22:09:48.400296 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:48.400164 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qz6gw" podUID="3b595cc6-bee1-476f-842c-00ed98dd3e79" Jul 28 22:09:49.398683 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:49.398648 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-djzl8" Jul 28 22:09:49.398867 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:49.398812 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-djzl8" podUID="59a1f798-88a8-4236-82f2-0a675d85920d" Jul 28 22:09:49.943012 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:49.942975 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m97d\" (UniqueName: \"kubernetes.io/projected/3b595cc6-bee1-476f-842c-00ed98dd3e79-kube-api-access-8m97d\") pod \"network-check-target-qz6gw\" (UID: \"3b595cc6-bee1-476f-842c-00ed98dd3e79\") " pod="openshift-network-diagnostics/network-check-target-qz6gw" Jul 28 22:09:49.943419 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:49.943065 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/59a1f798-88a8-4236-82f2-0a675d85920d-original-pull-secret\") pod \"global-pull-secret-syncer-djzl8\" (UID: \"59a1f798-88a8-4236-82f2-0a675d85920d\") " pod="kube-system/global-pull-secret-syncer-djzl8" Jul 28 22:09:49.943419 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:49.943181 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 22:09:49.943419 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:49.943243 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59a1f798-88a8-4236-82f2-0a675d85920d-original-pull-secret podName:59a1f798-88a8-4236-82f2-0a675d85920d nodeName:}" failed. No retries permitted until 2026-07-28 22:09:57.943218695 +0000 UTC m=+18.030476748 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/59a1f798-88a8-4236-82f2-0a675d85920d-original-pull-secret") pod "global-pull-secret-syncer-djzl8" (UID: "59a1f798-88a8-4236-82f2-0a675d85920d") : object "kube-system"/"original-pull-secret" not registered Jul 28 22:09:49.943419 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:49.943307 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 22:09:49.943419 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:49.943317 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 22:09:49.943419 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:49.943325 2553 projected.go:196] Error preparing data for projected volume kube-api-access-8m97d for pod openshift-network-diagnostics/network-check-target-qz6gw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 22:09:49.943419 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:49.943353 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b595cc6-bee1-476f-842c-00ed98dd3e79-kube-api-access-8m97d podName:3b595cc6-bee1-476f-842c-00ed98dd3e79 nodeName:}" failed. No retries permitted until 2026-07-28 22:09:57.943341745 +0000 UTC m=+18.030599795 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-8m97d" (UniqueName: "kubernetes.io/projected/3b595cc6-bee1-476f-842c-00ed98dd3e79-kube-api-access-8m97d") pod "network-check-target-qz6gw" (UID: "3b595cc6-bee1-476f-842c-00ed98dd3e79") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 22:09:50.044102 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:50.044076 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15e94249-e108-4426-b6f4-baf221c92a71-metrics-certs\") pod \"network-metrics-daemon-wl29s\" (UID: \"15e94249-e108-4426-b6f4-baf221c92a71\") " pod="openshift-multus/network-metrics-daemon-wl29s" Jul 28 22:09:50.044243 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:50.044204 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 22:09:50.044283 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:50.044247 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15e94249-e108-4426-b6f4-baf221c92a71-metrics-certs podName:15e94249-e108-4426-b6f4-baf221c92a71 nodeName:}" failed. No retries permitted until 2026-07-28 22:09:58.044234208 +0000 UTC m=+18.131492254 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/15e94249-e108-4426-b6f4-baf221c92a71-metrics-certs") pod "network-metrics-daemon-wl29s" (UID: "15e94249-e108-4426-b6f4-baf221c92a71") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 22:09:50.400088 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:50.400007 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qz6gw" Jul 28 22:09:50.400230 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:50.400124 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qz6gw" podUID="3b595cc6-bee1-476f-842c-00ed98dd3e79" Jul 28 22:09:50.400715 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:50.400528 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wl29s" Jul 28 22:09:50.400715 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:50.400648 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wl29s" podUID="15e94249-e108-4426-b6f4-baf221c92a71" Jul 28 22:09:51.398951 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:51.398916 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-djzl8" Jul 28 22:09:51.399408 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:51.399049 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-djzl8" podUID="59a1f798-88a8-4236-82f2-0a675d85920d" Jul 28 22:09:52.399401 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:52.398666 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wl29s" Jul 28 22:09:52.399401 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:52.398819 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wl29s" podUID="15e94249-e108-4426-b6f4-baf221c92a71" Jul 28 22:09:52.399401 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:52.399253 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qz6gw" Jul 28 22:09:52.399401 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:52.399357 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qz6gw" podUID="3b595cc6-bee1-476f-842c-00ed98dd3e79" Jul 28 22:09:53.398628 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:53.398593 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-djzl8" Jul 28 22:09:53.398806 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:53.398736 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-djzl8" podUID="59a1f798-88a8-4236-82f2-0a675d85920d" Jul 28 22:09:54.398775 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:54.398743 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wl29s" Jul 28 22:09:54.399229 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:54.398743 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qz6gw" Jul 28 22:09:54.399229 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:54.398888 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wl29s" podUID="15e94249-e108-4426-b6f4-baf221c92a71" Jul 28 22:09:54.399229 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:54.398952 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qz6gw" podUID="3b595cc6-bee1-476f-842c-00ed98dd3e79" Jul 28 22:09:55.398806 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:55.398777 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-djzl8" Jul 28 22:09:55.399255 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:55.398909 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-djzl8" podUID="59a1f798-88a8-4236-82f2-0a675d85920d" Jul 28 22:09:56.399664 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:56.399633 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wl29s" Jul 28 22:09:56.400138 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:56.399799 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wl29s" podUID="15e94249-e108-4426-b6f4-baf221c92a71" Jul 28 22:09:56.400138 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:56.399858 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qz6gw" Jul 28 22:09:56.400138 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:56.399962 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qz6gw" podUID="3b595cc6-bee1-476f-842c-00ed98dd3e79" Jul 28 22:09:57.399419 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:57.399387 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-djzl8" Jul 28 22:09:57.399617 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:57.399491 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-djzl8" podUID="59a1f798-88a8-4236-82f2-0a675d85920d" Jul 28 22:09:58.009883 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:58.009845 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/59a1f798-88a8-4236-82f2-0a675d85920d-original-pull-secret\") pod \"global-pull-secret-syncer-djzl8\" (UID: \"59a1f798-88a8-4236-82f2-0a675d85920d\") " pod="kube-system/global-pull-secret-syncer-djzl8" Jul 28 22:09:58.010440 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:58.009905 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m97d\" (UniqueName: \"kubernetes.io/projected/3b595cc6-bee1-476f-842c-00ed98dd3e79-kube-api-access-8m97d\") pod \"network-check-target-qz6gw\" (UID: \"3b595cc6-bee1-476f-842c-00ed98dd3e79\") " pod="openshift-network-diagnostics/network-check-target-qz6gw" Jul 28 22:09:58.010440 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:58.009978 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 22:09:58.010440 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:58.010052 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59a1f798-88a8-4236-82f2-0a675d85920d-original-pull-secret podName:59a1f798-88a8-4236-82f2-0a675d85920d nodeName:}" failed. No retries permitted until 2026-07-28 22:10:14.010034941 +0000 UTC m=+34.097292988 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/59a1f798-88a8-4236-82f2-0a675d85920d-original-pull-secret") pod "global-pull-secret-syncer-djzl8" (UID: "59a1f798-88a8-4236-82f2-0a675d85920d") : object "kube-system"/"original-pull-secret" not registered Jul 28 22:09:58.010440 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:58.010060 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 22:09:58.010440 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:58.010074 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 22:09:58.010440 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:58.010085 2553 projected.go:196] Error preparing data for projected volume kube-api-access-8m97d for pod openshift-network-diagnostics/network-check-target-qz6gw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 22:09:58.010440 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:58.010137 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b595cc6-bee1-476f-842c-00ed98dd3e79-kube-api-access-8m97d podName:3b595cc6-bee1-476f-842c-00ed98dd3e79 nodeName:}" failed. No retries permitted until 2026-07-28 22:10:14.010122837 +0000 UTC m=+34.097380885 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-8m97d" (UniqueName: "kubernetes.io/projected/3b595cc6-bee1-476f-842c-00ed98dd3e79-kube-api-access-8m97d") pod "network-check-target-qz6gw" (UID: "3b595cc6-bee1-476f-842c-00ed98dd3e79") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 22:09:58.110682 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:58.110648 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15e94249-e108-4426-b6f4-baf221c92a71-metrics-certs\") pod \"network-metrics-daemon-wl29s\" (UID: \"15e94249-e108-4426-b6f4-baf221c92a71\") " pod="openshift-multus/network-metrics-daemon-wl29s" Jul 28 22:09:58.110891 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:58.110803 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 22:09:58.110891 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:58.110866 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15e94249-e108-4426-b6f4-baf221c92a71-metrics-certs podName:15e94249-e108-4426-b6f4-baf221c92a71 nodeName:}" failed. No retries permitted until 2026-07-28 22:10:14.11084668 +0000 UTC m=+34.198104748 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/15e94249-e108-4426-b6f4-baf221c92a71-metrics-certs") pod "network-metrics-daemon-wl29s" (UID: "15e94249-e108-4426-b6f4-baf221c92a71") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 22:09:58.399096 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:58.399073 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wl29s" Jul 28 22:09:58.399200 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:58.399183 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wl29s" podUID="15e94249-e108-4426-b6f4-baf221c92a71" Jul 28 22:09:58.399248 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:58.399213 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qz6gw" Jul 28 22:09:58.399292 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:58.399278 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qz6gw" podUID="3b595cc6-bee1-476f-842c-00ed98dd3e79" Jul 28 22:09:59.398717 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:09:59.398668 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-djzl8" Jul 28 22:09:59.399136 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:09:59.398826 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-djzl8" podUID="59a1f798-88a8-4236-82f2-0a675d85920d" Jul 28 22:10:00.399883 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:00.399849 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qz6gw" Jul 28 22:10:00.401569 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:00.400364 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qz6gw" podUID="3b595cc6-bee1-476f-842c-00ed98dd3e79" Jul 28 22:10:00.401569 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:00.400450 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wl29s" Jul 28 22:10:00.401569 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:00.400620 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wl29s" podUID="15e94249-e108-4426-b6f4-baf221c92a71" Jul 28 22:10:01.398784 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:01.398741 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-djzl8" Jul 28 22:10:01.398971 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:01.398876 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-djzl8" podUID="59a1f798-88a8-4236-82f2-0a675d85920d" Jul 28 22:10:02.401935 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:02.400858 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wl29s" Jul 28 22:10:02.401935 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:02.401104 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qz6gw" Jul 28 22:10:02.401935 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:02.401218 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wl29s" podUID="15e94249-e108-4426-b6f4-baf221c92a71" Jul 28 22:10:02.401935 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:02.401493 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qz6gw" podUID="3b595cc6-bee1-476f-842c-00ed98dd3e79" Jul 28 22:10:02.504098 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:02.504059 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6k2jz" event={"ID":"935749df-a4bd-4733-9085-1f8d1e4aa86a","Type":"ContainerStarted","Data":"00d536bec5e5838be55aa8f228b536bf457fea3a7c090d8aaabc903e31fdf6a3"} Jul 28 22:10:02.506629 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:02.506547 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bs64q" event={"ID":"54dc150c-a1f7-4df4-9adf-03077bf59073","Type":"ContainerStarted","Data":"89a10ea2ddb48f2c842d5c756eb5ff71d346fd7f7ac9b06a5d04f8ab33dcb025"} Jul 28 22:10:02.508294 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:02.508255 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zcxg2" event={"ID":"327f1776-1215-4c6e-9fef-3e30c12b1b66","Type":"ContainerStarted","Data":"4d804a5b6326fc4a058d82e23abedba57dc9444c9e68157d4ccb83627bd45fc9"} Jul 28 22:10:02.510056 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:02.509874 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7q942" event={"ID":"c93df2e7-d79a-4a98-8556-a98868dfd93e","Type":"ContainerStarted","Data":"7e31cce6b4a8b7d884c16a61fc4377bbd1deb7a5b3aec0cdcf0a70d7835959b9"} Jul 28 22:10:02.512011 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:02.511730 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbzq5" event={"ID":"a55216ee-cd69-4f12-af05-1d12a69e7c93","Type":"ContainerStarted","Data":"cf28652a24578b7658f8e927ee41dc8a11ef48ab83d2cad9c137356a4126ee8b"} Jul 28 22:10:02.514120 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:02.514089 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-w9bnn" event={"ID":"0f57ff96-5ed8-4a73-9dfd-7e5d45bd163c","Type":"ContainerStarted","Data":"9cb2c6c7ed7332fbd496406a4fd4d1fc420235c39c0b2f943a7f891f7484c14c"} Jul 28 22:10:02.515679 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:02.515645 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" event={"ID":"842e457b-ef75-43bc-a182-e3abca66995a","Type":"ContainerStarted","Data":"92cb90733a8b50c743bf645e1a4e90fe6ddf4e44cd32c29b1b734a2563f2c205"} Jul 28 22:10:02.517116 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:02.516985 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xvmx5" event={"ID":"74555be4-d49b-4d9c-9e80-c4604c40cd23","Type":"ContainerStarted","Data":"2b0d06ca1005ecc4666968f27f3d95db4f7547972e8074cc5837c013aabb007d"} Jul 28 22:10:02.520380 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:02.520343 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-132.ec2.internal" podStartSLOduration=21.520330446 podStartE2EDuration="21.520330446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 22:09:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:09:45.476660462 +0000 UTC m=+5.563918538" watchObservedRunningTime="2026-07-28 22:10:02.520330446 +0000 UTC m=+22.607588515" Jul 28 22:10:02.520758 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:02.520733 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-6k2jz" podStartSLOduration=3.5471295659999997 podStartE2EDuration="22.520724848s" totalImagesPullingTime="18.973595282s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 22:09:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:10:02.520124987 +0000 UTC m=+22.607383057" watchObservedRunningTime="2026-07-28 22:10:02.520724848 +0000 UTC m=+22.607982909" Jul 28 22:10:02.535024 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:02.534856 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-w9bnn" podStartSLOduration=3.562719256 podStartE2EDuration="22.534836187s" totalImagesPullingTime="18.972116931s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 22:09:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:10:02.534299902 +0000 UTC m=+22.621557960" watchObservedRunningTime="2026-07-28 22:10:02.534836187 +0000 UTC m=+22.622094257" Jul 28 22:10:02.564502 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:02.564310 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-xvmx5" podStartSLOduration=3.534989864 podStartE2EDuration="22.56429345s" totalImagesPullingTime="19.029303586s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 22:09:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:10:02.563227024 +0000 UTC m=+22.650485094" watchObservedRunningTime="2026-07-28 22:10:02.56429345 +0000 UTC m=+22.651551519" Jul 28 22:10:02.588442 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:02.588389 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bs64q" podStartSLOduration=3.632673977 podStartE2EDuration="22.588375339s" totalImagesPullingTime="18.955701362s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 22:09:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:10:02.587961101 +0000 UTC m=+22.675219167" watchObservedRunningTime="2026-07-28 22:10:02.588375339 +0000 UTC m=+22.675633407" Jul 28 22:10:02.629058 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:02.629012 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-7q942" podStartSLOduration=7.499122629 podStartE2EDuration="22.62899454s" totalImagesPullingTime="15.129871911s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:09:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:10:02.628596978 +0000 UTC m=+22.715855047" watchObservedRunningTime="2026-07-28 22:10:02.62899454 +0000 UTC m=+22.716252608" Jul 28 22:10:03.398593 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:03.398561 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-djzl8" Jul 28 22:10:03.398764 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:03.398667 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-djzl8" podUID="59a1f798-88a8-4236-82f2-0a675d85920d" Jul 28 22:10:03.520010 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:03.519978 2553 generic.go:356] "Generic (PLEG): container finished" podID="327f1776-1215-4c6e-9fef-3e30c12b1b66" containerID="4d804a5b6326fc4a058d82e23abedba57dc9444c9e68157d4ccb83627bd45fc9" exitCode=0 Jul 28 22:10:03.520721 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:03.520070 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zcxg2" event={"ID":"327f1776-1215-4c6e-9fef-3e30c12b1b66","Type":"ContainerDied","Data":"4d804a5b6326fc4a058d82e23abedba57dc9444c9e68157d4ccb83627bd45fc9"} Jul 28 22:10:03.523155 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:03.523130 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" event={"ID":"842e457b-ef75-43bc-a182-e3abca66995a","Type":"ContainerStarted","Data":"b0ab05d406113aa149ac3fa98719719617b9024aecb780454de463b3892fab31"} Jul 28 22:10:03.523270 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:03.523219 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" event={"ID":"842e457b-ef75-43bc-a182-e3abca66995a","Type":"ContainerStarted","Data":"ce27bbe6477bce13b7b7bab00f2be1eca1e688a3585a45d153453a3ba84012e5"} Jul 28 22:10:03.523315 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:03.523286 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" event={"ID":"842e457b-ef75-43bc-a182-e3abca66995a","Type":"ContainerStarted","Data":"81bb42d2fcd8d9926f36190c8595d779817204a436d20a87e00de55284d6a989"} Jul 28 22:10:03.523349 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:03.523338 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" event={"ID":"842e457b-ef75-43bc-a182-e3abca66995a","Type":"ContainerStarted","Data":"7f98cf73f243351871ffaf6fcaebcd13d4f8470cd6d0a335e47eac1e10bf8c67"} Jul 28 22:10:03.523408 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:03.523393 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" event={"ID":"842e457b-ef75-43bc-a182-e3abca66995a","Type":"ContainerStarted","Data":"f6e6906e9d9e7849ae6372262047e5844e825ed44581b4f334823abff268bb3f"} Jul 28 22:10:03.860269 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:03.860234 2553 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 28 22:10:04.339461 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:04.339346 2553 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T22:10:03.860252933Z","UUID":"e249e6d1-b5e7-4fc6-96fe-65dc64ba66ef","Handler":null,"Name":"","Endpoint":""} Jul 28 22:10:04.341389 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:04.341338 2553 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 28 22:10:04.341389 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:04.341377 2553 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 28 22:10:04.399071 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:04.399040 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qz6gw" Jul 28 22:10:04.399263 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:04.399175 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qz6gw" podUID="3b595cc6-bee1-476f-842c-00ed98dd3e79" Jul 28 22:10:04.399263 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:04.399205 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wl29s" Jul 28 22:10:04.399388 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:04.399277 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wl29s" podUID="15e94249-e108-4426-b6f4-baf221c92a71" Jul 28 22:10:04.527240 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:04.527197 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-flt69" event={"ID":"cb4e193d-e952-43fb-9161-f45d39dd37ec","Type":"ContainerStarted","Data":"24075f96164fbc2e4387a32b1e42a4dd4132842db25130003006a3abaa22c19e"} Jul 28 22:10:04.529350 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:04.529320 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbzq5" event={"ID":"a55216ee-cd69-4f12-af05-1d12a69e7c93","Type":"ContainerStarted","Data":"54fba17126110b44abb7047780216c3897b81bc46b261b13a3148d7854b17ede"} Jul 28 22:10:04.678344 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:04.678263 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-7q942" Jul 28 22:10:04.678926 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:04.678902 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-7q942" Jul 28 22:10:04.695169 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:04.695112 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-flt69" podStartSLOduration=5.757022185 podStartE2EDuration="24.695090636s" totalImagesPullingTime="18.938068451s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 22:09:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:10:04.542193324 +0000 UTC m=+24.629451393" watchObservedRunningTime="2026-07-28 22:10:04.695090636 +0000 UTC m=+24.782348706" Jul 28 22:10:05.399365 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:05.399334 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-djzl8" Jul 28 22:10:05.399569 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:05.399464 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-djzl8" podUID="59a1f798-88a8-4236-82f2-0a675d85920d" Jul 28 22:10:05.535109 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:05.534330 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbzq5" event={"ID":"a55216ee-cd69-4f12-af05-1d12a69e7c93","Type":"ContainerStarted","Data":"5fd017de98520c826c497f9be67ac1c44022eedf2f3a3f15e6b372d1b87545f0"} Jul 28 22:10:05.539295 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:05.539263 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" event={"ID":"842e457b-ef75-43bc-a182-e3abca66995a","Type":"ContainerStarted","Data":"e00dff05f4b39812aff0f0eeed8d1b162b352224dde510974b0e87aea7f6f010"} Jul 28 22:10:05.539490 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:05.539473 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-7q942" Jul 28 22:10:05.539959 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:05.539940 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7q942" Jul 28 22:10:05.552898 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:05.552861 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fbzq5" podStartSLOduration=4.2681773849999995 podStartE2EDuration="25.552847311s" totalImagesPullingTime="21.284669926s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 22:09:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:10:05.552471774 +0000 UTC m=+25.639729843" watchObservedRunningTime="2026-07-28 22:10:05.552847311 +0000 UTC m=+25.640105379" Jul 28 22:10:06.399370 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:06.399152 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qz6gw" Jul 28 22:10:06.399567 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:06.399183 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wl29s" Jul 28 22:10:06.399567 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:06.399489 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qz6gw" podUID="3b595cc6-bee1-476f-842c-00ed98dd3e79" Jul 28 22:10:06.399567 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:06.399519 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wl29s" podUID="15e94249-e108-4426-b6f4-baf221c92a71" Jul 28 22:10:07.399595 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:07.399561 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-djzl8" Jul 28 22:10:07.400186 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:07.399713 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-djzl8" podUID="59a1f798-88a8-4236-82f2-0a675d85920d" Jul 28 22:10:08.399749 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:08.399573 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wl29s" Jul 28 22:10:08.400206 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:08.399633 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qz6gw" Jul 28 22:10:08.400206 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:08.399840 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wl29s" podUID="15e94249-e108-4426-b6f4-baf221c92a71" Jul 28 22:10:08.400206 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:08.399923 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qz6gw" podUID="3b595cc6-bee1-476f-842c-00ed98dd3e79" Jul 28 22:10:08.549127 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:08.549095 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" event={"ID":"842e457b-ef75-43bc-a182-e3abca66995a","Type":"ContainerStarted","Data":"185df61d40457ca88ce0ab9ca1a7c7a0350ba19bd99cf91a9ebb47bd948f88f4"} Jul 28 22:10:08.549367 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:08.549339 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" Jul 28 22:10:08.550952 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:08.550930 2553 generic.go:356] "Generic (PLEG): container finished" podID="327f1776-1215-4c6e-9fef-3e30c12b1b66" containerID="ccea41bc5f49a27b0f73a067c2745cc1fefae427a80671e017f6d6cccfee8e16" exitCode=0 Jul 28 22:10:08.551056 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:08.550964 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zcxg2" event={"ID":"327f1776-1215-4c6e-9fef-3e30c12b1b66","Type":"ContainerDied","Data":"ccea41bc5f49a27b0f73a067c2745cc1fefae427a80671e017f6d6cccfee8e16"} Jul 28 22:10:08.565130 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:08.565111 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" Jul 28 22:10:08.573636 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:08.573597 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" podStartSLOduration=9.558680627 podStartE2EDuration="28.573583531s" totalImagesPullingTime="19.014902904s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 22:09:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:10:08.573575201 +0000 UTC m=+28.660833307" watchObservedRunningTime="2026-07-28 22:10:08.573583531 +0000 UTC m=+28.660841597" Jul 28 22:10:09.398748 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:09.398720 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-djzl8" Jul 28 22:10:09.398962 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:09.398820 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-djzl8" podUID="59a1f798-88a8-4236-82f2-0a675d85920d" Jul 28 22:10:09.553199 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:09.553171 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" Jul 28 22:10:09.553490 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:09.553242 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" Jul 28 22:10:09.578587 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:09.578566 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" Jul 28 22:10:10.149488 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:10.149457 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qz6gw"] Jul 28 22:10:10.149648 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:10.149611 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qz6gw" Jul 28 22:10:10.149727 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:10.149690 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qz6gw" podUID="3b595cc6-bee1-476f-842c-00ed98dd3e79" Jul 28 22:10:10.152387 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:10.152354 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wl29s"] Jul 28 22:10:10.152660 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:10.152482 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wl29s" Jul 28 22:10:10.152660 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:10.152581 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wl29s" podUID="15e94249-e108-4426-b6f4-baf221c92a71" Jul 28 22:10:10.165316 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:10.165288 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-djzl8"] Jul 28 22:10:10.165420 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:10.165385 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-djzl8" Jul 28 22:10:10.165490 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:10.165470 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-djzl8" podUID="59a1f798-88a8-4236-82f2-0a675d85920d" Jul 28 22:10:10.557108 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:10.557073 2553 generic.go:356] "Generic (PLEG): container finished" podID="327f1776-1215-4c6e-9fef-3e30c12b1b66" containerID="750f021b8fdc53d340ad496bb6dcc25e5c667f601c691f72ff86d0a3c2653332" exitCode=0 Jul 28 22:10:10.557590 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:10.557167 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zcxg2" event={"ID":"327f1776-1215-4c6e-9fef-3e30c12b1b66","Type":"ContainerDied","Data":"750f021b8fdc53d340ad496bb6dcc25e5c667f601c691f72ff86d0a3c2653332"} Jul 28 22:10:11.399610 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:11.399580 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wl29s" Jul 28 22:10:11.399771 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:11.399749 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wl29s" podUID="15e94249-e108-4426-b6f4-baf221c92a71" Jul 28 22:10:12.399531 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:12.399303 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qz6gw" Jul 28 22:10:12.399993 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:12.399357 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-djzl8" Jul 28 22:10:12.399993 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:12.399586 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qz6gw" podUID="3b595cc6-bee1-476f-842c-00ed98dd3e79" Jul 28 22:10:12.399993 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:12.399626 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-djzl8" podUID="59a1f798-88a8-4236-82f2-0a675d85920d" Jul 28 22:10:12.562933 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:12.562899 2553 generic.go:356] "Generic (PLEG): container finished" podID="327f1776-1215-4c6e-9fef-3e30c12b1b66" containerID="a9def559f528d5d2efc10b1363bd5a8b15872c9057893e2e76848fa71d7611f1" exitCode=0 Jul 28 22:10:12.563099 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:12.562938 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zcxg2" event={"ID":"327f1776-1215-4c6e-9fef-3e30c12b1b66","Type":"ContainerDied","Data":"a9def559f528d5d2efc10b1363bd5a8b15872c9057893e2e76848fa71d7611f1"} Jul 28 22:10:13.399402 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:13.399365 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wl29s" Jul 28 22:10:13.399598 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:13.399521 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wl29s" podUID="15e94249-e108-4426-b6f4-baf221c92a71" Jul 28 22:10:14.027032 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:14.026935 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/59a1f798-88a8-4236-82f2-0a675d85920d-original-pull-secret\") pod \"global-pull-secret-syncer-djzl8\" (UID: \"59a1f798-88a8-4236-82f2-0a675d85920d\") " pod="kube-system/global-pull-secret-syncer-djzl8" Jul 28 22:10:14.027032 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:14.026989 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m97d\" (UniqueName: \"kubernetes.io/projected/3b595cc6-bee1-476f-842c-00ed98dd3e79-kube-api-access-8m97d\") pod \"network-check-target-qz6gw\" (UID: \"3b595cc6-bee1-476f-842c-00ed98dd3e79\") " pod="openshift-network-diagnostics/network-check-target-qz6gw" Jul 28 22:10:14.027267 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:14.027097 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 22:10:14.027267 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:14.027236 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59a1f798-88a8-4236-82f2-0a675d85920d-original-pull-secret podName:59a1f798-88a8-4236-82f2-0a675d85920d nodeName:}" failed. No retries permitted until 2026-07-28 22:10:46.027209026 +0000 UTC m=+66.114467088 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/59a1f798-88a8-4236-82f2-0a675d85920d-original-pull-secret") pod "global-pull-secret-syncer-djzl8" (UID: "59a1f798-88a8-4236-82f2-0a675d85920d") : object "kube-system"/"original-pull-secret" not registered Jul 28 22:10:14.027267 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:14.027112 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 22:10:14.027432 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:14.027285 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 22:10:14.027432 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:14.027299 2553 projected.go:196] Error preparing data for projected volume kube-api-access-8m97d for pod openshift-network-diagnostics/network-check-target-qz6gw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 22:10:14.027432 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:14.027354 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b595cc6-bee1-476f-842c-00ed98dd3e79-kube-api-access-8m97d podName:3b595cc6-bee1-476f-842c-00ed98dd3e79 nodeName:}" failed. No retries permitted until 2026-07-28 22:10:46.027338627 +0000 UTC m=+66.114596673 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-8m97d" (UniqueName: "kubernetes.io/projected/3b595cc6-bee1-476f-842c-00ed98dd3e79-kube-api-access-8m97d") pod "network-check-target-qz6gw" (UID: "3b595cc6-bee1-476f-842c-00ed98dd3e79") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 22:10:14.128294 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:14.128257 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15e94249-e108-4426-b6f4-baf221c92a71-metrics-certs\") pod \"network-metrics-daemon-wl29s\" (UID: \"15e94249-e108-4426-b6f4-baf221c92a71\") " pod="openshift-multus/network-metrics-daemon-wl29s" Jul 28 22:10:14.128466 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:14.128441 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 22:10:14.128524 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:14.128517 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15e94249-e108-4426-b6f4-baf221c92a71-metrics-certs podName:15e94249-e108-4426-b6f4-baf221c92a71 nodeName:}" failed. No retries permitted until 2026-07-28 22:10:46.128496037 +0000 UTC m=+66.215754101 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/15e94249-e108-4426-b6f4-baf221c92a71-metrics-certs") pod "network-metrics-daemon-wl29s" (UID: "15e94249-e108-4426-b6f4-baf221c92a71") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 22:10:14.398884 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:14.398795 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qz6gw" Jul 28 22:10:14.398884 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:14.398848 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-djzl8" Jul 28 22:10:14.399101 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:14.398927 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qz6gw" podUID="3b595cc6-bee1-476f-842c-00ed98dd3e79" Jul 28 22:10:14.399101 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:14.399076 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-djzl8" podUID="59a1f798-88a8-4236-82f2-0a675d85920d" Jul 28 22:10:15.253669 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.253637 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-132.ec2.internal" event="NodeReady" Jul 28 22:10:15.254288 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.253793 2553 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 28 22:10:15.287247 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.287203 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-7mp2z"] Jul 28 22:10:15.289118 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.289097 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-7mp2z" Jul 28 22:10:15.302119 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.302084 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 28 22:10:15.302365 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.302332 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 28 22:10:15.303042 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.302149 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-65465d7954-dqhtb"] Jul 28 22:10:15.303042 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.303030 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-4dvmv\"" Jul 28 22:10:15.303181 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.302987 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 28 22:10:15.304001 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.303980 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 28 22:10:15.305994 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.305972 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-ncbpn"] Jul 28 22:10:15.306214 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.306195 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65465d7954-dqhtb" Jul 28 22:10:15.307376 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.307357 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-zn5fm"] Jul 28 22:10:15.309029 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.307637 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-ncbpn" Jul 28 22:10:15.309942 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.309920 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-mldhp"] Jul 28 22:10:15.311236 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.311198 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 28 22:10:15.311351 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.311332 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hf5kg\"" Jul 28 22:10:15.311351 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.311342 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 28 22:10:15.311463 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.311383 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-4rjdf\"" Jul 28 22:10:15.311463 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.311435 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 28 22:10:15.312139 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.311624 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-zn5fm" Jul 28 22:10:15.312139 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.311787 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 28 22:10:15.312139 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.311828 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-2dpdh"] Jul 28 22:10:15.312139 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.311833 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 28 22:10:15.312430 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.312264 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-mldhp" Jul 28 22:10:15.313442 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.313425 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-bpt84"] Jul 28 22:10:15.313943 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.313928 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-2dpdh" Jul 28 22:10:15.315403 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.315128 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 28 22:10:15.315521 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.315493 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-8nnqd"] Jul 28 22:10:15.315615 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.315550 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-bpt84" Jul 28 22:10:15.318064 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.317474 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-5k4gd\"" Jul 28 22:10:15.318180 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.318165 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 28 22:10:15.318237 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.318172 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 28 22:10:15.318767 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.318436 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 28 22:10:15.318767 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.318452 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-2vjxd\"" Jul 28 22:10:15.318767 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.318750 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 28 22:10:15.319275 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.319117 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 28 22:10:15.319275 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.319249 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 28 22:10:15.319545 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.319529 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-8bb56\"" Jul 28 22:10:15.319842 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.319821 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 28 22:10:15.320836 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.320816 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-gmlz2\"" Jul 28 22:10:15.321037 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.321018 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 28 22:10:15.321278 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.321258 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 28 22:10:15.321363 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.321304 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 28 22:10:15.321572 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.321553 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 28 22:10:15.321749 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.321260 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 28 22:10:15.322942 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.322142 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 28 22:10:15.324870 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.324850 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 28 22:10:15.325883 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.325859 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6747b7576c-fs2bd"] Jul 28 22:10:15.326814 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.326097 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-8nnqd" Jul 28 22:10:15.327477 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.327451 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 28 22:10:15.329483 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.329464 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 28 22:10:15.329990 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.329972 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nzswh"] Jul 28 22:10:15.333472 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.330591 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-8wgb6\"" Jul 28 22:10:15.333472 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.331172 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 28 22:10:15.333472 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.331299 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 28 22:10:15.336362 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.335530 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 28 22:10:15.336362 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.336170 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6747b7576c-fs2bd" Jul 28 22:10:15.337367 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.337347 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-7mp2z"] Jul 28 22:10:15.337460 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.337390 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-bgl6d"] Jul 28 22:10:15.337644 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.337629 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nzswh" Jul 28 22:10:15.339663 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.339645 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 28 22:10:15.339917 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.339815 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 28 22:10:15.339917 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.339754 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 28 22:10:15.340254 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.340233 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 28 22:10:15.340254 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.340237 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 28 22:10:15.340384 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.340298 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 28 22:10:15.340606 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.340589 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 28 22:10:15.346516 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.340601 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-8dvgg\"" Jul 28 22:10:15.346516 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.340600 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-xrrc8\"" Jul 28 22:10:15.346516 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.340816 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 28 22:10:15.346516 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.340605 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 28 22:10:15.346516 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.340813 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-bgl6d" Jul 28 22:10:15.346516 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.340659 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-ncbpn"] Jul 28 22:10:15.346516 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.341073 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-2dpdh"] Jul 28 22:10:15.346516 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.341079 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 28 22:10:15.346516 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.341112 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-zn5fm"] Jul 28 22:10:15.346516 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.341133 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-8nnqd"] Jul 28 22:10:15.346516 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.341155 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-mldhp"] Jul 28 22:10:15.346516 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.341172 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-65465d7954-dqhtb"] Jul 28 22:10:15.346516 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.341192 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6747b7576c-fs2bd"] Jul 28 22:10:15.346516 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.341209 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-bgl6d"] Jul 28 22:10:15.346516 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.341413 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-l6gf5"] Jul 28 22:10:15.347342 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.347290 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-q8wn6"] Jul 28 22:10:15.348185 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.347593 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-l6gf5" Jul 28 22:10:15.348507 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.348480 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-dr6qj\"" Jul 28 22:10:15.348858 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.348837 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 28 22:10:15.349181 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.349159 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 28 22:10:15.350023 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.350005 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 28 22:10:15.350129 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.350011 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 28 22:10:15.350530 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.350393 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vcmq5\"" Jul 28 22:10:15.351241 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.351211 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nzswh"] Jul 28 22:10:15.351323 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.351299 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-l6gf5"] Jul 28 22:10:15.351395 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.351329 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-q8wn6" Jul 28 22:10:15.351684 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.351558 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-bpt84"] Jul 28 22:10:15.351684 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.351596 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-q8wn6"] Jul 28 22:10:15.353276 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.353260 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 28 22:10:15.353424 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.353409 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-lsdr6\"" Jul 28 22:10:15.353619 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.353604 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 28 22:10:15.353799 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.353774 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 28 22:10:15.398926 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.398903 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wl29s" Jul 28 22:10:15.401565 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.401550 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 28 22:10:15.403611 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.403587 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-q7z5n\"" Jul 28 22:10:15.437297 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.437268 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/414a9396-e46b-433a-9976-95c5c0b1d897-snapshots\") pod \"insights-operator-5885fdb759-bpt84\" (UID: \"414a9396-e46b-433a-9976-95c5c0b1d897\") " pod="openshift-insights/insights-operator-5885fdb759-bpt84" Jul 28 22:10:15.437409 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.437313 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/414a9396-e46b-433a-9976-95c5c0b1d897-service-ca-bundle\") pod \"insights-operator-5885fdb759-bpt84\" (UID: \"414a9396-e46b-433a-9976-95c5c0b1d897\") " pod="openshift-insights/insights-operator-5885fdb759-bpt84" Jul 28 22:10:15.437409 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.437359 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eed632c4-ac4f-4f76-8a0e-a12bcf00835d-trusted-ca\") pod \"image-registry-65465d7954-dqhtb\" (UID: \"eed632c4-ac4f-4f76-8a0e-a12bcf00835d\") " pod="openshift-image-registry/image-registry-65465d7954-dqhtb" Jul 28 22:10:15.437409 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.437384 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8cea718f-40c0-48b7-943c-7c4178bd8d0d-cert\") pod \"ingress-canary-q8wn6\" (UID: \"8cea718f-40c0-48b7-943c-7c4178bd8d0d\") " pod="openshift-ingress-canary/ingress-canary-q8wn6" Jul 28 22:10:15.437409 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.437405 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/eb1d24ec-acde-48ed-9004-aa372bc53057-config-volume\") pod \"dns-default-l6gf5\" (UID: \"eb1d24ec-acde-48ed-9004-aa372bc53057\") " pod="openshift-dns/dns-default-l6gf5" Jul 28 22:10:15.437585 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.437428 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e1ced996-f136-4d3e-9c85-a5b7809b4517-config\") pod \"console-operator-584dcb6b54-mldhp\" (UID: \"e1ced996-f136-4d3e-9c85-a5b7809b4517\") " pod="openshift-console-operator/console-operator-584dcb6b54-mldhp" Jul 28 22:10:15.437585 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.437482 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e1d395ef-80a9-4851-ad04-f6377dd12795-serving-cert\") pod \"service-ca-operator-5d6c75f84f-7mp2z\" (UID: \"e1d395ef-80a9-4851-ad04-f6377dd12795\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-7mp2z" Jul 28 22:10:15.437585 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.437538 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/eed632c4-ac4f-4f76-8a0e-a12bcf00835d-registry-certificates\") pod \"image-registry-65465d7954-dqhtb\" (UID: \"eed632c4-ac4f-4f76-8a0e-a12bcf00835d\") " pod="openshift-image-registry/image-registry-65465d7954-dqhtb" Jul 28 22:10:15.437727 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.437584 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt7wt\" (UniqueName: \"kubernetes.io/projected/139cae0a-1747-4e84-b210-24cac8e94541-kube-api-access-zt7wt\") pod \"cluster-monitoring-operator-548b489cb8-zn5fm\" (UID: \"139cae0a-1747-4e84-b210-24cac8e94541\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-zn5fm" Jul 28 22:10:15.437727 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.437609 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c1aff3e5-115b-4502-899a-b2137168973b-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-nzswh\" (UID: \"c1aff3e5-115b-4502-899a-b2137168973b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nzswh" Jul 28 22:10:15.437727 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.437624 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7e266916-99eb-4807-9b85-fabdeb9e1c28-default-certificate\") pod \"router-default-6747b7576c-fs2bd\" (UID: \"7e266916-99eb-4807-9b85-fabdeb9e1c28\") " pod="openshift-ingress/router-default-6747b7576c-fs2bd" Jul 28 22:10:15.437727 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.437638 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eb1d24ec-acde-48ed-9004-aa372bc53057-metrics-tls\") pod \"dns-default-l6gf5\" (UID: \"eb1d24ec-acde-48ed-9004-aa372bc53057\") " pod="openshift-dns/dns-default-l6gf5" Jul 28 22:10:15.437727 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.437657 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgb67\" (UniqueName: \"kubernetes.io/projected/eed632c4-ac4f-4f76-8a0e-a12bcf00835d-kube-api-access-xgb67\") pod \"image-registry-65465d7954-dqhtb\" (UID: \"eed632c4-ac4f-4f76-8a0e-a12bcf00835d\") " pod="openshift-image-registry/image-registry-65465d7954-dqhtb" Jul 28 22:10:15.437727 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.437671 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5v9z\" (UniqueName: \"kubernetes.io/projected/eb1d24ec-acde-48ed-9004-aa372bc53057-kube-api-access-w5v9z\") pod \"dns-default-l6gf5\" (UID: \"eb1d24ec-acde-48ed-9004-aa372bc53057\") " pod="openshift-dns/dns-default-l6gf5" Jul 28 22:10:15.437727 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.437685 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/414a9396-e46b-433a-9976-95c5c0b1d897-tmp\") pod \"insights-operator-5885fdb759-bpt84\" (UID: \"414a9396-e46b-433a-9976-95c5c0b1d897\") " pod="openshift-insights/insights-operator-5885fdb759-bpt84" Jul 28 22:10:15.437727 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.437717 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/414a9396-e46b-433a-9976-95c5c0b1d897-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-bpt84\" (UID: \"414a9396-e46b-433a-9976-95c5c0b1d897\") " pod="openshift-insights/insights-operator-5885fdb759-bpt84" Jul 28 22:10:15.438115 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.437768 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4529840b-b684-4b37-b9ab-b066f7456f1b-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-ncbpn\" (UID: \"4529840b-b684-4b37-b9ab-b066f7456f1b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-ncbpn" Jul 28 22:10:15.438115 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.437799 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb1d24ec-acde-48ed-9004-aa372bc53057-tmp-dir\") pod \"dns-default-l6gf5\" (UID: \"eb1d24ec-acde-48ed-9004-aa372bc53057\") " pod="openshift-dns/dns-default-l6gf5" Jul 28 22:10:15.438115 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.437826 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/139cae0a-1747-4e84-b210-24cac8e94541-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-zn5fm\" (UID: \"139cae0a-1747-4e84-b210-24cac8e94541\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-zn5fm" Jul 28 22:10:15.438115 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.437868 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e266916-99eb-4807-9b85-fabdeb9e1c28-metrics-certs\") pod \"router-default-6747b7576c-fs2bd\" (UID: \"7e266916-99eb-4807-9b85-fabdeb9e1c28\") " pod="openshift-ingress/router-default-6747b7576c-fs2bd" Jul 28 22:10:15.438115 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.437912 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/eed632c4-ac4f-4f76-8a0e-a12bcf00835d-image-registry-private-configuration\") pod \"image-registry-65465d7954-dqhtb\" (UID: \"eed632c4-ac4f-4f76-8a0e-a12bcf00835d\") " pod="openshift-image-registry/image-registry-65465d7954-dqhtb" Jul 28 22:10:15.438115 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.437937 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/eed632c4-ac4f-4f76-8a0e-a12bcf00835d-installation-pull-secrets\") pod \"image-registry-65465d7954-dqhtb\" (UID: \"eed632c4-ac4f-4f76-8a0e-a12bcf00835d\") " pod="openshift-image-registry/image-registry-65465d7954-dqhtb" Jul 28 22:10:15.438115 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.437960 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hddc4\" (UniqueName: \"kubernetes.io/projected/8cea718f-40c0-48b7-943c-7c4178bd8d0d-kube-api-access-hddc4\") pod \"ingress-canary-q8wn6\" (UID: \"8cea718f-40c0-48b7-943c-7c4178bd8d0d\") " pod="openshift-ingress-canary/ingress-canary-q8wn6" Jul 28 22:10:15.438115 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.437986 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qmvs\" (UniqueName: \"kubernetes.io/projected/7e266916-99eb-4807-9b85-fabdeb9e1c28-kube-api-access-9qmvs\") pod \"router-default-6747b7576c-fs2bd\" (UID: \"7e266916-99eb-4807-9b85-fabdeb9e1c28\") " pod="openshift-ingress/router-default-6747b7576c-fs2bd" Jul 28 22:10:15.438115 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.438012 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/414a9396-e46b-433a-9976-95c5c0b1d897-serving-cert\") pod \"insights-operator-5885fdb759-bpt84\" (UID: \"414a9396-e46b-433a-9976-95c5c0b1d897\") " pod="openshift-insights/insights-operator-5885fdb759-bpt84" Jul 28 22:10:15.438115 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.438035 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e1d395ef-80a9-4851-ad04-f6377dd12795-config\") pod \"service-ca-operator-5d6c75f84f-7mp2z\" (UID: \"e1d395ef-80a9-4851-ad04-f6377dd12795\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-7mp2z" Jul 28 22:10:15.438115 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.438058 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/eed632c4-ac4f-4f76-8a0e-a12bcf00835d-ca-trust-extracted\") pod \"image-registry-65465d7954-dqhtb\" (UID: \"eed632c4-ac4f-4f76-8a0e-a12bcf00835d\") " pod="openshift-image-registry/image-registry-65465d7954-dqhtb" Jul 28 22:10:15.438115 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.438083 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drdc6\" (UniqueName: \"kubernetes.io/projected/b2f1e639-88f4-4c61-96ae-3a7b694e44e1-kube-api-access-drdc6\") pod \"volume-data-source-validator-549c6d54cf-2dpdh\" (UID: \"b2f1e639-88f4-4c61-96ae-3a7b694e44e1\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-2dpdh" Jul 28 22:10:15.438546 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.438154 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58djb\" (UniqueName: \"kubernetes.io/projected/c1aff3e5-115b-4502-899a-b2137168973b-kube-api-access-58djb\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-nzswh\" (UID: \"c1aff3e5-115b-4502-899a-b2137168973b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nzswh" Jul 28 22:10:15.438546 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.438179 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4529840b-b684-4b37-b9ab-b066f7456f1b-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-ncbpn\" (UID: \"4529840b-b684-4b37-b9ab-b066f7456f1b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-ncbpn" Jul 28 22:10:15.438546 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.438212 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e1ced996-f136-4d3e-9c85-a5b7809b4517-serving-cert\") pod \"console-operator-584dcb6b54-mldhp\" (UID: \"e1ced996-f136-4d3e-9c85-a5b7809b4517\") " pod="openshift-console-operator/console-operator-584dcb6b54-mldhp" Jul 28 22:10:15.438546 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.438242 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7e266916-99eb-4807-9b85-fabdeb9e1c28-stats-auth\") pod \"router-default-6747b7576c-fs2bd\" (UID: \"7e266916-99eb-4807-9b85-fabdeb9e1c28\") " pod="openshift-ingress/router-default-6747b7576c-fs2bd" Jul 28 22:10:15.438546 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.438266 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr8gt\" (UniqueName: \"kubernetes.io/projected/48d47548-1ab9-4983-8e36-9de23b54b911-kube-api-access-rr8gt\") pod \"network-check-source-7d88cb55d8-bgl6d\" (UID: \"48d47548-1ab9-4983-8e36-9de23b54b911\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-bgl6d" Jul 28 22:10:15.438546 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.438287 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4cdq\" (UniqueName: \"kubernetes.io/projected/414a9396-e46b-433a-9976-95c5c0b1d897-kube-api-access-q4cdq\") pod \"insights-operator-5885fdb759-bpt84\" (UID: \"414a9396-e46b-433a-9976-95c5c0b1d897\") " pod="openshift-insights/insights-operator-5885fdb759-bpt84" Jul 28 22:10:15.438546 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.438308 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eed632c4-ac4f-4f76-8a0e-a12bcf00835d-registry-tls\") pod \"image-registry-65465d7954-dqhtb\" (UID: \"eed632c4-ac4f-4f76-8a0e-a12bcf00835d\") " pod="openshift-image-registry/image-registry-65465d7954-dqhtb" Jul 28 22:10:15.438546 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.438323 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e1ced996-f136-4d3e-9c85-a5b7809b4517-trusted-ca\") pod \"console-operator-584dcb6b54-mldhp\" (UID: \"e1ced996-f136-4d3e-9c85-a5b7809b4517\") " pod="openshift-console-operator/console-operator-584dcb6b54-mldhp" Jul 28 22:10:15.438546 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.438350 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eed632c4-ac4f-4f76-8a0e-a12bcf00835d-bound-sa-token\") pod \"image-registry-65465d7954-dqhtb\" (UID: \"eed632c4-ac4f-4f76-8a0e-a12bcf00835d\") " pod="openshift-image-registry/image-registry-65465d7954-dqhtb" Jul 28 22:10:15.438546 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.438378 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c1aff3e5-115b-4502-899a-b2137168973b-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-nzswh\" (UID: \"c1aff3e5-115b-4502-899a-b2137168973b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nzswh" Jul 28 22:10:15.438546 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.438417 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a2d6df48-f09b-4f10-b909-dedec2d5b3d0-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-8nnqd\" (UID: \"a2d6df48-f09b-4f10-b909-dedec2d5b3d0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-8nnqd" Jul 28 22:10:15.438546 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.438473 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4vfz\" (UniqueName: \"kubernetes.io/projected/a2d6df48-f09b-4f10-b909-dedec2d5b3d0-kube-api-access-d4vfz\") pod \"cluster-samples-operator-6748489656-8nnqd\" (UID: \"a2d6df48-f09b-4f10-b909-dedec2d5b3d0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-8nnqd" Jul 28 22:10:15.438546 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.438509 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7vf9\" (UniqueName: \"kubernetes.io/projected/e1d395ef-80a9-4851-ad04-f6377dd12795-kube-api-access-m7vf9\") pod \"service-ca-operator-5d6c75f84f-7mp2z\" (UID: \"e1d395ef-80a9-4851-ad04-f6377dd12795\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-7mp2z" Jul 28 22:10:15.438546 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.438533 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/139cae0a-1747-4e84-b210-24cac8e94541-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-zn5fm\" (UID: \"139cae0a-1747-4e84-b210-24cac8e94541\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-zn5fm" Jul 28 22:10:15.438986 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.438557 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e266916-99eb-4807-9b85-fabdeb9e1c28-service-ca-bundle\") pod \"router-default-6747b7576c-fs2bd\" (UID: \"7e266916-99eb-4807-9b85-fabdeb9e1c28\") " pod="openshift-ingress/router-default-6747b7576c-fs2bd" Jul 28 22:10:15.438986 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.438581 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhz82\" (UniqueName: \"kubernetes.io/projected/e1ced996-f136-4d3e-9c85-a5b7809b4517-kube-api-access-hhz82\") pod \"console-operator-584dcb6b54-mldhp\" (UID: \"e1ced996-f136-4d3e-9c85-a5b7809b4517\") " pod="openshift-console-operator/console-operator-584dcb6b54-mldhp" Jul 28 22:10:15.489566 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.489531 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7q942_c93df2e7-d79a-4a98-8556-a98868dfd93e/konnectivity-agent/0.log" limit=1024 Jul 28 22:10:15.539811 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.539736 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/414a9396-e46b-433a-9976-95c5c0b1d897-snapshots\") pod \"insights-operator-5885fdb759-bpt84\" (UID: \"414a9396-e46b-433a-9976-95c5c0b1d897\") " pod="openshift-insights/insights-operator-5885fdb759-bpt84" Jul 28 22:10:15.539811 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.539769 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/414a9396-e46b-433a-9976-95c5c0b1d897-service-ca-bundle\") pod \"insights-operator-5885fdb759-bpt84\" (UID: \"414a9396-e46b-433a-9976-95c5c0b1d897\") " pod="openshift-insights/insights-operator-5885fdb759-bpt84" Jul 28 22:10:15.539811 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.539799 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eed632c4-ac4f-4f76-8a0e-a12bcf00835d-trusted-ca\") pod \"image-registry-65465d7954-dqhtb\" (UID: \"eed632c4-ac4f-4f76-8a0e-a12bcf00835d\") " pod="openshift-image-registry/image-registry-65465d7954-dqhtb" Jul 28 22:10:15.540052 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.539889 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8cea718f-40c0-48b7-943c-7c4178bd8d0d-cert\") pod \"ingress-canary-q8wn6\" (UID: \"8cea718f-40c0-48b7-943c-7c4178bd8d0d\") " pod="openshift-ingress-canary/ingress-canary-q8wn6" Jul 28 22:10:15.540052 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.540003 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/eb1d24ec-acde-48ed-9004-aa372bc53057-config-volume\") pod \"dns-default-l6gf5\" (UID: \"eb1d24ec-acde-48ed-9004-aa372bc53057\") " pod="openshift-dns/dns-default-l6gf5" Jul 28 22:10:15.540052 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.540042 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e1ced996-f136-4d3e-9c85-a5b7809b4517-config\") pod \"console-operator-584dcb6b54-mldhp\" (UID: \"e1ced996-f136-4d3e-9c85-a5b7809b4517\") " pod="openshift-console-operator/console-operator-584dcb6b54-mldhp" Jul 28 22:10:15.540214 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.540070 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e1d395ef-80a9-4851-ad04-f6377dd12795-serving-cert\") pod \"service-ca-operator-5d6c75f84f-7mp2z\" (UID: \"e1d395ef-80a9-4851-ad04-f6377dd12795\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-7mp2z" Jul 28 22:10:15.540214 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.540095 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/eed632c4-ac4f-4f76-8a0e-a12bcf00835d-registry-certificates\") pod \"image-registry-65465d7954-dqhtb\" (UID: \"eed632c4-ac4f-4f76-8a0e-a12bcf00835d\") " pod="openshift-image-registry/image-registry-65465d7954-dqhtb" Jul 28 22:10:15.540214 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.540123 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt7wt\" (UniqueName: \"kubernetes.io/projected/139cae0a-1747-4e84-b210-24cac8e94541-kube-api-access-zt7wt\") pod \"cluster-monitoring-operator-548b489cb8-zn5fm\" (UID: \"139cae0a-1747-4e84-b210-24cac8e94541\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-zn5fm" Jul 28 22:10:15.540214 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.540148 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c1aff3e5-115b-4502-899a-b2137168973b-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-nzswh\" (UID: \"c1aff3e5-115b-4502-899a-b2137168973b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nzswh" Jul 28 22:10:15.540214 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.540175 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7e266916-99eb-4807-9b85-fabdeb9e1c28-default-certificate\") pod \"router-default-6747b7576c-fs2bd\" (UID: \"7e266916-99eb-4807-9b85-fabdeb9e1c28\") " pod="openshift-ingress/router-default-6747b7576c-fs2bd" Jul 28 22:10:15.540214 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.540202 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eb1d24ec-acde-48ed-9004-aa372bc53057-metrics-tls\") pod \"dns-default-l6gf5\" (UID: \"eb1d24ec-acde-48ed-9004-aa372bc53057\") " pod="openshift-dns/dns-default-l6gf5" Jul 28 22:10:15.540510 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.540229 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgb67\" (UniqueName: \"kubernetes.io/projected/eed632c4-ac4f-4f76-8a0e-a12bcf00835d-kube-api-access-xgb67\") pod \"image-registry-65465d7954-dqhtb\" (UID: \"eed632c4-ac4f-4f76-8a0e-a12bcf00835d\") " pod="openshift-image-registry/image-registry-65465d7954-dqhtb" Jul 28 22:10:15.540510 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.540253 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5v9z\" (UniqueName: \"kubernetes.io/projected/eb1d24ec-acde-48ed-9004-aa372bc53057-kube-api-access-w5v9z\") pod \"dns-default-l6gf5\" (UID: \"eb1d24ec-acde-48ed-9004-aa372bc53057\") " pod="openshift-dns/dns-default-l6gf5" Jul 28 22:10:15.540510 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.540281 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/414a9396-e46b-433a-9976-95c5c0b1d897-tmp\") pod \"insights-operator-5885fdb759-bpt84\" (UID: \"414a9396-e46b-433a-9976-95c5c0b1d897\") " pod="openshift-insights/insights-operator-5885fdb759-bpt84" Jul 28 22:10:15.540510 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.540306 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/414a9396-e46b-433a-9976-95c5c0b1d897-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-bpt84\" (UID: \"414a9396-e46b-433a-9976-95c5c0b1d897\") " pod="openshift-insights/insights-operator-5885fdb759-bpt84" Jul 28 22:10:15.540510 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.540334 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4529840b-b684-4b37-b9ab-b066f7456f1b-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-ncbpn\" (UID: \"4529840b-b684-4b37-b9ab-b066f7456f1b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-ncbpn" Jul 28 22:10:15.540510 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.540360 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb1d24ec-acde-48ed-9004-aa372bc53057-tmp-dir\") pod \"dns-default-l6gf5\" (UID: \"eb1d24ec-acde-48ed-9004-aa372bc53057\") " pod="openshift-dns/dns-default-l6gf5" Jul 28 22:10:15.540510 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.540389 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/139cae0a-1747-4e84-b210-24cac8e94541-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-zn5fm\" (UID: \"139cae0a-1747-4e84-b210-24cac8e94541\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-zn5fm" Jul 28 22:10:15.540510 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.540413 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e266916-99eb-4807-9b85-fabdeb9e1c28-metrics-certs\") pod \"router-default-6747b7576c-fs2bd\" (UID: \"7e266916-99eb-4807-9b85-fabdeb9e1c28\") " pod="openshift-ingress/router-default-6747b7576c-fs2bd" Jul 28 22:10:15.540510 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.540457 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/eed632c4-ac4f-4f76-8a0e-a12bcf00835d-image-registry-private-configuration\") pod \"image-registry-65465d7954-dqhtb\" (UID: \"eed632c4-ac4f-4f76-8a0e-a12bcf00835d\") " pod="openshift-image-registry/image-registry-65465d7954-dqhtb" Jul 28 22:10:15.540510 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.540464 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/414a9396-e46b-433a-9976-95c5c0b1d897-service-ca-bundle\") pod \"insights-operator-5885fdb759-bpt84\" (UID: \"414a9396-e46b-433a-9976-95c5c0b1d897\") " pod="openshift-insights/insights-operator-5885fdb759-bpt84" Jul 28 22:10:15.540510 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.540487 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/eed632c4-ac4f-4f76-8a0e-a12bcf00835d-installation-pull-secrets\") pod \"image-registry-65465d7954-dqhtb\" (UID: \"eed632c4-ac4f-4f76-8a0e-a12bcf00835d\") " pod="openshift-image-registry/image-registry-65465d7954-dqhtb" Jul 28 22:10:15.540510 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.540513 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hddc4\" (UniqueName: \"kubernetes.io/projected/8cea718f-40c0-48b7-943c-7c4178bd8d0d-kube-api-access-hddc4\") pod \"ingress-canary-q8wn6\" (UID: \"8cea718f-40c0-48b7-943c-7c4178bd8d0d\") " pod="openshift-ingress-canary/ingress-canary-q8wn6" Jul 28 22:10:15.541050 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.540538 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qmvs\" (UniqueName: \"kubernetes.io/projected/7e266916-99eb-4807-9b85-fabdeb9e1c28-kube-api-access-9qmvs\") pod \"router-default-6747b7576c-fs2bd\" (UID: \"7e266916-99eb-4807-9b85-fabdeb9e1c28\") " pod="openshift-ingress/router-default-6747b7576c-fs2bd" Jul 28 22:10:15.541050 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.540568 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/414a9396-e46b-433a-9976-95c5c0b1d897-serving-cert\") pod \"insights-operator-5885fdb759-bpt84\" (UID: \"414a9396-e46b-433a-9976-95c5c0b1d897\") " pod="openshift-insights/insights-operator-5885fdb759-bpt84" Jul 28 22:10:15.541050 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.540597 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e1d395ef-80a9-4851-ad04-f6377dd12795-config\") pod \"service-ca-operator-5d6c75f84f-7mp2z\" (UID: \"e1d395ef-80a9-4851-ad04-f6377dd12795\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-7mp2z" Jul 28 22:10:15.541050 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.540622 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/eed632c4-ac4f-4f76-8a0e-a12bcf00835d-ca-trust-extracted\") pod \"image-registry-65465d7954-dqhtb\" (UID: \"eed632c4-ac4f-4f76-8a0e-a12bcf00835d\") " pod="openshift-image-registry/image-registry-65465d7954-dqhtb" Jul 28 22:10:15.541050 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.540646 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drdc6\" (UniqueName: \"kubernetes.io/projected/b2f1e639-88f4-4c61-96ae-3a7b694e44e1-kube-api-access-drdc6\") pod \"volume-data-source-validator-549c6d54cf-2dpdh\" (UID: \"b2f1e639-88f4-4c61-96ae-3a7b694e44e1\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-2dpdh" Jul 28 22:10:15.541050 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.540738 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58djb\" (UniqueName: \"kubernetes.io/projected/c1aff3e5-115b-4502-899a-b2137168973b-kube-api-access-58djb\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-nzswh\" (UID: \"c1aff3e5-115b-4502-899a-b2137168973b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nzswh" Jul 28 22:10:15.541050 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.540769 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4529840b-b684-4b37-b9ab-b066f7456f1b-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-ncbpn\" (UID: \"4529840b-b684-4b37-b9ab-b066f7456f1b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-ncbpn" Jul 28 22:10:15.541050 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.540803 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e1ced996-f136-4d3e-9c85-a5b7809b4517-serving-cert\") pod \"console-operator-584dcb6b54-mldhp\" (UID: \"e1ced996-f136-4d3e-9c85-a5b7809b4517\") " pod="openshift-console-operator/console-operator-584dcb6b54-mldhp" Jul 28 22:10:15.541050 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.540839 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7e266916-99eb-4807-9b85-fabdeb9e1c28-stats-auth\") pod \"router-default-6747b7576c-fs2bd\" (UID: \"7e266916-99eb-4807-9b85-fabdeb9e1c28\") " pod="openshift-ingress/router-default-6747b7576c-fs2bd" Jul 28 22:10:15.541050 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.540864 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr8gt\" (UniqueName: \"kubernetes.io/projected/48d47548-1ab9-4983-8e36-9de23b54b911-kube-api-access-rr8gt\") pod \"network-check-source-7d88cb55d8-bgl6d\" (UID: \"48d47548-1ab9-4983-8e36-9de23b54b911\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-bgl6d" Jul 28 22:10:15.541050 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.540890 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4cdq\" (UniqueName: \"kubernetes.io/projected/414a9396-e46b-433a-9976-95c5c0b1d897-kube-api-access-q4cdq\") pod \"insights-operator-5885fdb759-bpt84\" (UID: \"414a9396-e46b-433a-9976-95c5c0b1d897\") " pod="openshift-insights/insights-operator-5885fdb759-bpt84" Jul 28 22:10:15.541050 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.540918 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eed632c4-ac4f-4f76-8a0e-a12bcf00835d-registry-tls\") pod \"image-registry-65465d7954-dqhtb\" (UID: \"eed632c4-ac4f-4f76-8a0e-a12bcf00835d\") " pod="openshift-image-registry/image-registry-65465d7954-dqhtb" Jul 28 22:10:15.541050 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.540933 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eed632c4-ac4f-4f76-8a0e-a12bcf00835d-trusted-ca\") pod \"image-registry-65465d7954-dqhtb\" (UID: \"eed632c4-ac4f-4f76-8a0e-a12bcf00835d\") " pod="openshift-image-registry/image-registry-65465d7954-dqhtb" Jul 28 22:10:15.541050 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.540945 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e1ced996-f136-4d3e-9c85-a5b7809b4517-trusted-ca\") pod \"console-operator-584dcb6b54-mldhp\" (UID: \"e1ced996-f136-4d3e-9c85-a5b7809b4517\") " pod="openshift-console-operator/console-operator-584dcb6b54-mldhp" Jul 28 22:10:15.541050 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.540980 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eed632c4-ac4f-4f76-8a0e-a12bcf00835d-bound-sa-token\") pod \"image-registry-65465d7954-dqhtb\" (UID: \"eed632c4-ac4f-4f76-8a0e-a12bcf00835d\") " pod="openshift-image-registry/image-registry-65465d7954-dqhtb" Jul 28 22:10:15.541796 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.541014 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c1aff3e5-115b-4502-899a-b2137168973b-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-nzswh\" (UID: \"c1aff3e5-115b-4502-899a-b2137168973b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nzswh" Jul 28 22:10:15.541796 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.541062 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a2d6df48-f09b-4f10-b909-dedec2d5b3d0-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-8nnqd\" (UID: \"a2d6df48-f09b-4f10-b909-dedec2d5b3d0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-8nnqd" Jul 28 22:10:15.541796 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.541100 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4vfz\" (UniqueName: \"kubernetes.io/projected/a2d6df48-f09b-4f10-b909-dedec2d5b3d0-kube-api-access-d4vfz\") pod \"cluster-samples-operator-6748489656-8nnqd\" (UID: \"a2d6df48-f09b-4f10-b909-dedec2d5b3d0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-8nnqd" Jul 28 22:10:15.541796 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.541113 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/eb1d24ec-acde-48ed-9004-aa372bc53057-config-volume\") pod \"dns-default-l6gf5\" (UID: \"eb1d24ec-acde-48ed-9004-aa372bc53057\") " pod="openshift-dns/dns-default-l6gf5" Jul 28 22:10:15.541796 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.541131 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7vf9\" (UniqueName: \"kubernetes.io/projected/e1d395ef-80a9-4851-ad04-f6377dd12795-kube-api-access-m7vf9\") pod \"service-ca-operator-5d6c75f84f-7mp2z\" (UID: \"e1d395ef-80a9-4851-ad04-f6377dd12795\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-7mp2z" Jul 28 22:10:15.541796 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.541187 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/139cae0a-1747-4e84-b210-24cac8e94541-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-zn5fm\" (UID: \"139cae0a-1747-4e84-b210-24cac8e94541\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-zn5fm" Jul 28 22:10:15.541796 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.541220 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e266916-99eb-4807-9b85-fabdeb9e1c28-service-ca-bundle\") pod \"router-default-6747b7576c-fs2bd\" (UID: \"7e266916-99eb-4807-9b85-fabdeb9e1c28\") " pod="openshift-ingress/router-default-6747b7576c-fs2bd" Jul 28 22:10:15.541796 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.541247 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhz82\" (UniqueName: \"kubernetes.io/projected/e1ced996-f136-4d3e-9c85-a5b7809b4517-kube-api-access-hhz82\") pod \"console-operator-584dcb6b54-mldhp\" (UID: \"e1ced996-f136-4d3e-9c85-a5b7809b4517\") " pod="openshift-console-operator/console-operator-584dcb6b54-mldhp" Jul 28 22:10:15.541796 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.541554 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/414a9396-e46b-433a-9976-95c5c0b1d897-snapshots\") pod \"insights-operator-5885fdb759-bpt84\" (UID: \"414a9396-e46b-433a-9976-95c5c0b1d897\") " pod="openshift-insights/insights-operator-5885fdb759-bpt84" Jul 28 22:10:15.541796 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.541600 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e1ced996-f136-4d3e-9c85-a5b7809b4517-config\") pod \"console-operator-584dcb6b54-mldhp\" (UID: \"e1ced996-f136-4d3e-9c85-a5b7809b4517\") " pod="openshift-console-operator/console-operator-584dcb6b54-mldhp" Jul 28 22:10:15.541796 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.541757 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e1ced996-f136-4d3e-9c85-a5b7809b4517-trusted-ca\") pod \"console-operator-584dcb6b54-mldhp\" (UID: \"e1ced996-f136-4d3e-9c85-a5b7809b4517\") " pod="openshift-console-operator/console-operator-584dcb6b54-mldhp" Jul 28 22:10:15.542286 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.542127 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c1aff3e5-115b-4502-899a-b2137168973b-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-nzswh\" (UID: \"c1aff3e5-115b-4502-899a-b2137168973b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nzswh" Jul 28 22:10:15.542286 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:15.542217 2553 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 28 22:10:15.542286 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:15.542273 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2d6df48-f09b-4f10-b909-dedec2d5b3d0-samples-operator-tls podName:a2d6df48-f09b-4f10-b909-dedec2d5b3d0 nodeName:}" failed. No retries permitted until 2026-07-28 22:10:16.042254849 +0000 UTC m=+36.129512912 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a2d6df48-f09b-4f10-b909-dedec2d5b3d0-samples-operator-tls") pod "cluster-samples-operator-6748489656-8nnqd" (UID: "a2d6df48-f09b-4f10-b909-dedec2d5b3d0") : secret "samples-operator-tls" not found Jul 28 22:10:15.542461 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.542434 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e1d395ef-80a9-4851-ad04-f6377dd12795-config\") pod \"service-ca-operator-5d6c75f84f-7mp2z\" (UID: \"e1d395ef-80a9-4851-ad04-f6377dd12795\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-7mp2z" Jul 28 22:10:15.542878 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.542827 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/414a9396-e46b-433a-9976-95c5c0b1d897-tmp\") pod \"insights-operator-5885fdb759-bpt84\" (UID: \"414a9396-e46b-433a-9976-95c5c0b1d897\") " pod="openshift-insights/insights-operator-5885fdb759-bpt84" Jul 28 22:10:15.542978 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.542877 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/eed632c4-ac4f-4f76-8a0e-a12bcf00835d-registry-certificates\") pod \"image-registry-65465d7954-dqhtb\" (UID: \"eed632c4-ac4f-4f76-8a0e-a12bcf00835d\") " pod="openshift-image-registry/image-registry-65465d7954-dqhtb" Jul 28 22:10:15.543313 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.543293 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/139cae0a-1747-4e84-b210-24cac8e94541-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-zn5fm\" (UID: \"139cae0a-1747-4e84-b210-24cac8e94541\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-zn5fm" Jul 28 22:10:15.543415 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:15.543406 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7e266916-99eb-4807-9b85-fabdeb9e1c28-service-ca-bundle podName:7e266916-99eb-4807-9b85-fabdeb9e1c28 nodeName:}" failed. No retries permitted until 2026-07-28 22:10:16.043389864 +0000 UTC m=+36.130647938 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7e266916-99eb-4807-9b85-fabdeb9e1c28-service-ca-bundle") pod "router-default-6747b7576c-fs2bd" (UID: "7e266916-99eb-4807-9b85-fabdeb9e1c28") : configmap references non-existent config key: service-ca.crt Jul 28 22:10:15.544305 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.543490 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/eed632c4-ac4f-4f76-8a0e-a12bcf00835d-ca-trust-extracted\") pod \"image-registry-65465d7954-dqhtb\" (UID: \"eed632c4-ac4f-4f76-8a0e-a12bcf00835d\") " pod="openshift-image-registry/image-registry-65465d7954-dqhtb" Jul 28 22:10:15.544305 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.543546 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4529840b-b684-4b37-b9ab-b066f7456f1b-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-ncbpn\" (UID: \"4529840b-b684-4b37-b9ab-b066f7456f1b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-ncbpn" Jul 28 22:10:15.544305 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.543734 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/414a9396-e46b-433a-9976-95c5c0b1d897-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-bpt84\" (UID: \"414a9396-e46b-433a-9976-95c5c0b1d897\") " pod="openshift-insights/insights-operator-5885fdb759-bpt84" Jul 28 22:10:15.544305 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:15.543777 2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 22:10:15.544305 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.543797 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb1d24ec-acde-48ed-9004-aa372bc53057-tmp-dir\") pod \"dns-default-l6gf5\" (UID: \"eb1d24ec-acde-48ed-9004-aa372bc53057\") " pod="openshift-dns/dns-default-l6gf5" Jul 28 22:10:15.544305 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:15.543823 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb1d24ec-acde-48ed-9004-aa372bc53057-metrics-tls podName:eb1d24ec-acde-48ed-9004-aa372bc53057 nodeName:}" failed. No retries permitted until 2026-07-28 22:10:16.043810503 +0000 UTC m=+36.131068549 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/eb1d24ec-acde-48ed-9004-aa372bc53057-metrics-tls") pod "dns-default-l6gf5" (UID: "eb1d24ec-acde-48ed-9004-aa372bc53057") : secret "dns-default-metrics-tls" not found Jul 28 22:10:15.544305 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:15.543829 2553 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 28 22:10:15.544305 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:15.543890 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/139cae0a-1747-4e84-b210-24cac8e94541-cluster-monitoring-operator-tls podName:139cae0a-1747-4e84-b210-24cac8e94541 nodeName:}" failed. No retries permitted until 2026-07-28 22:10:16.043873687 +0000 UTC m=+36.131131753 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/139cae0a-1747-4e84-b210-24cac8e94541-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-zn5fm" (UID: "139cae0a-1747-4e84-b210-24cac8e94541") : secret "cluster-monitoring-operator-tls" not found Jul 28 22:10:15.544305 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:15.543922 2553 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 22:10:15.544305 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:15.543939 2553 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65465d7954-dqhtb: secret "image-registry-tls" not found Jul 28 22:10:15.544305 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:15.543992 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eed632c4-ac4f-4f76-8a0e-a12bcf00835d-registry-tls podName:eed632c4-ac4f-4f76-8a0e-a12bcf00835d nodeName:}" failed. No retries permitted until 2026-07-28 22:10:16.043977558 +0000 UTC m=+36.131235606 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/eed632c4-ac4f-4f76-8a0e-a12bcf00835d-registry-tls") pod "image-registry-65465d7954-dqhtb" (UID: "eed632c4-ac4f-4f76-8a0e-a12bcf00835d") : secret "image-registry-tls" not found Jul 28 22:10:15.544305 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:15.544151 2553 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 28 22:10:15.544305 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:15.544205 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4529840b-b684-4b37-b9ab-b066f7456f1b-networking-console-plugin-cert podName:4529840b-b684-4b37-b9ab-b066f7456f1b nodeName:}" failed. No retries permitted until 2026-07-28 22:10:16.044191772 +0000 UTC m=+36.131449837 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/4529840b-b684-4b37-b9ab-b066f7456f1b-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-ncbpn" (UID: "4529840b-b684-4b37-b9ab-b066f7456f1b") : secret "networking-console-plugin-cert" not found Jul 28 22:10:15.545016 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:15.544551 2553 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 28 22:10:15.545016 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:15.544605 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e266916-99eb-4807-9b85-fabdeb9e1c28-metrics-certs podName:7e266916-99eb-4807-9b85-fabdeb9e1c28 nodeName:}" failed. No retries permitted until 2026-07-28 22:10:16.044591538 +0000 UTC m=+36.131849591 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7e266916-99eb-4807-9b85-fabdeb9e1c28-metrics-certs") pod "router-default-6747b7576c-fs2bd" (UID: "7e266916-99eb-4807-9b85-fabdeb9e1c28") : secret "router-metrics-certs-default" not found Jul 28 22:10:15.546221 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.545764 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e1d395ef-80a9-4851-ad04-f6377dd12795-serving-cert\") pod \"service-ca-operator-5d6c75f84f-7mp2z\" (UID: \"e1d395ef-80a9-4851-ad04-f6377dd12795\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-7mp2z" Jul 28 22:10:15.546221 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.546216 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8cea718f-40c0-48b7-943c-7c4178bd8d0d-cert\") pod \"ingress-canary-q8wn6\" (UID: \"8cea718f-40c0-48b7-943c-7c4178bd8d0d\") " pod="openshift-ingress-canary/ingress-canary-q8wn6" Jul 28 22:10:15.546424 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.546398 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7e266916-99eb-4807-9b85-fabdeb9e1c28-default-certificate\") pod \"router-default-6747b7576c-fs2bd\" (UID: \"7e266916-99eb-4807-9b85-fabdeb9e1c28\") " pod="openshift-ingress/router-default-6747b7576c-fs2bd" Jul 28 22:10:15.546955 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.546936 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/414a9396-e46b-433a-9976-95c5c0b1d897-serving-cert\") pod \"insights-operator-5885fdb759-bpt84\" (UID: \"414a9396-e46b-433a-9976-95c5c0b1d897\") " pod="openshift-insights/insights-operator-5885fdb759-bpt84" Jul 28 22:10:15.547252 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.547223 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/eed632c4-ac4f-4f76-8a0e-a12bcf00835d-installation-pull-secrets\") pod \"image-registry-65465d7954-dqhtb\" (UID: \"eed632c4-ac4f-4f76-8a0e-a12bcf00835d\") " pod="openshift-image-registry/image-registry-65465d7954-dqhtb" Jul 28 22:10:15.548062 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.548032 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/eed632c4-ac4f-4f76-8a0e-a12bcf00835d-image-registry-private-configuration\") pod \"image-registry-65465d7954-dqhtb\" (UID: \"eed632c4-ac4f-4f76-8a0e-a12bcf00835d\") " pod="openshift-image-registry/image-registry-65465d7954-dqhtb" Jul 28 22:10:15.548264 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.548240 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c1aff3e5-115b-4502-899a-b2137168973b-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-nzswh\" (UID: \"c1aff3e5-115b-4502-899a-b2137168973b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nzswh" Jul 28 22:10:15.548625 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.548607 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e1ced996-f136-4d3e-9c85-a5b7809b4517-serving-cert\") pod \"console-operator-584dcb6b54-mldhp\" (UID: \"e1ced996-f136-4d3e-9c85-a5b7809b4517\") " pod="openshift-console-operator/console-operator-584dcb6b54-mldhp" Jul 28 22:10:15.549302 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.549285 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7e266916-99eb-4807-9b85-fabdeb9e1c28-stats-auth\") pod \"router-default-6747b7576c-fs2bd\" (UID: \"7e266916-99eb-4807-9b85-fabdeb9e1c28\") " pod="openshift-ingress/router-default-6747b7576c-fs2bd" Jul 28 22:10:15.557347 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.557123 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5v9z\" (UniqueName: \"kubernetes.io/projected/eb1d24ec-acde-48ed-9004-aa372bc53057-kube-api-access-w5v9z\") pod \"dns-default-l6gf5\" (UID: \"eb1d24ec-acde-48ed-9004-aa372bc53057\") " pod="openshift-dns/dns-default-l6gf5" Jul 28 22:10:15.558362 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.558336 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt7wt\" (UniqueName: \"kubernetes.io/projected/139cae0a-1747-4e84-b210-24cac8e94541-kube-api-access-zt7wt\") pod \"cluster-monitoring-operator-548b489cb8-zn5fm\" (UID: \"139cae0a-1747-4e84-b210-24cac8e94541\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-zn5fm" Jul 28 22:10:15.560894 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.560868 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hddc4\" (UniqueName: \"kubernetes.io/projected/8cea718f-40c0-48b7-943c-7c4178bd8d0d-kube-api-access-hddc4\") pod \"ingress-canary-q8wn6\" (UID: \"8cea718f-40c0-48b7-943c-7c4178bd8d0d\") " pod="openshift-ingress-canary/ingress-canary-q8wn6" Jul 28 22:10:15.562062 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.562019 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4cdq\" (UniqueName: \"kubernetes.io/projected/414a9396-e46b-433a-9976-95c5c0b1d897-kube-api-access-q4cdq\") pod \"insights-operator-5885fdb759-bpt84\" (UID: \"414a9396-e46b-433a-9976-95c5c0b1d897\") " pod="openshift-insights/insights-operator-5885fdb759-bpt84" Jul 28 22:10:15.562394 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.562372 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr8gt\" (UniqueName: \"kubernetes.io/projected/48d47548-1ab9-4983-8e36-9de23b54b911-kube-api-access-rr8gt\") pod \"network-check-source-7d88cb55d8-bgl6d\" (UID: \"48d47548-1ab9-4983-8e36-9de23b54b911\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-bgl6d" Jul 28 22:10:15.562483 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.562424 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7vf9\" (UniqueName: \"kubernetes.io/projected/e1d395ef-80a9-4851-ad04-f6377dd12795-kube-api-access-m7vf9\") pod \"service-ca-operator-5d6c75f84f-7mp2z\" (UID: \"e1d395ef-80a9-4851-ad04-f6377dd12795\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-7mp2z" Jul 28 22:10:15.562601 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.562578 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drdc6\" (UniqueName: \"kubernetes.io/projected/b2f1e639-88f4-4c61-96ae-3a7b694e44e1-kube-api-access-drdc6\") pod \"volume-data-source-validator-549c6d54cf-2dpdh\" (UID: \"b2f1e639-88f4-4c61-96ae-3a7b694e44e1\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-2dpdh" Jul 28 22:10:15.563315 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.563296 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eed632c4-ac4f-4f76-8a0e-a12bcf00835d-bound-sa-token\") pod \"image-registry-65465d7954-dqhtb\" (UID: \"eed632c4-ac4f-4f76-8a0e-a12bcf00835d\") " pod="openshift-image-registry/image-registry-65465d7954-dqhtb" Jul 28 22:10:15.563405 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.563345 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4vfz\" (UniqueName: \"kubernetes.io/projected/a2d6df48-f09b-4f10-b909-dedec2d5b3d0-kube-api-access-d4vfz\") pod \"cluster-samples-operator-6748489656-8nnqd\" (UID: \"a2d6df48-f09b-4f10-b909-dedec2d5b3d0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-8nnqd" Jul 28 22:10:15.563785 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.563723 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58djb\" (UniqueName: \"kubernetes.io/projected/c1aff3e5-115b-4502-899a-b2137168973b-kube-api-access-58djb\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-nzswh\" (UID: \"c1aff3e5-115b-4502-899a-b2137168973b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nzswh" Jul 28 22:10:15.564332 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.564310 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qmvs\" (UniqueName: \"kubernetes.io/projected/7e266916-99eb-4807-9b85-fabdeb9e1c28-kube-api-access-9qmvs\") pod \"router-default-6747b7576c-fs2bd\" (UID: \"7e266916-99eb-4807-9b85-fabdeb9e1c28\") " pod="openshift-ingress/router-default-6747b7576c-fs2bd" Jul 28 22:10:15.564397 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.564315 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhz82\" (UniqueName: \"kubernetes.io/projected/e1ced996-f136-4d3e-9c85-a5b7809b4517-kube-api-access-hhz82\") pod \"console-operator-584dcb6b54-mldhp\" (UID: \"e1ced996-f136-4d3e-9c85-a5b7809b4517\") " pod="openshift-console-operator/console-operator-584dcb6b54-mldhp" Jul 28 22:10:15.566535 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.566506 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgb67\" (UniqueName: \"kubernetes.io/projected/eed632c4-ac4f-4f76-8a0e-a12bcf00835d-kube-api-access-xgb67\") pod \"image-registry-65465d7954-dqhtb\" (UID: \"eed632c4-ac4f-4f76-8a0e-a12bcf00835d\") " pod="openshift-image-registry/image-registry-65465d7954-dqhtb" Jul 28 22:10:15.572566 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.572540 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7q942_c93df2e7-d79a-4a98-8556-a98868dfd93e/konnectivity-agent/0.log" limit=1024 Jul 28 22:10:15.615389 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.615355 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-7mp2z" Jul 28 22:10:15.659842 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.659816 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-mldhp" Jul 28 22:10:15.667471 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.667451 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-2dpdh" Jul 28 22:10:15.675020 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.674997 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-bpt84" Jul 28 22:10:15.696887 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.696855 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nzswh" Jul 28 22:10:15.703384 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.703363 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-bgl6d" Jul 28 22:10:15.718994 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.718974 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-q8wn6" Jul 28 22:10:15.781457 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:15.781418 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7q942_c93df2e7-d79a-4a98-8556-a98868dfd93e/konnectivity-agent/0.log" limit=1024 Jul 28 22:10:16.046441 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:16.046411 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4529840b-b684-4b37-b9ab-b066f7456f1b-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-ncbpn\" (UID: \"4529840b-b684-4b37-b9ab-b066f7456f1b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-ncbpn" Jul 28 22:10:16.046604 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:16.046455 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eed632c4-ac4f-4f76-8a0e-a12bcf00835d-registry-tls\") pod \"image-registry-65465d7954-dqhtb\" (UID: \"eed632c4-ac4f-4f76-8a0e-a12bcf00835d\") " pod="openshift-image-registry/image-registry-65465d7954-dqhtb" Jul 28 22:10:16.046604 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:16.046493 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a2d6df48-f09b-4f10-b909-dedec2d5b3d0-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-8nnqd\" (UID: \"a2d6df48-f09b-4f10-b909-dedec2d5b3d0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-8nnqd" Jul 28 22:10:16.046604 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:16.046527 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e266916-99eb-4807-9b85-fabdeb9e1c28-service-ca-bundle\") pod \"router-default-6747b7576c-fs2bd\" (UID: \"7e266916-99eb-4807-9b85-fabdeb9e1c28\") " pod="openshift-ingress/router-default-6747b7576c-fs2bd" Jul 28 22:10:16.046604 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:16.046581 2553 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 28 22:10:16.046778 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:16.046630 2553 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 28 22:10:16.046778 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:16.046632 2553 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 22:10:16.046778 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:16.046654 2553 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65465d7954-dqhtb: secret "image-registry-tls" not found Jul 28 22:10:16.046778 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:16.046655 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4529840b-b684-4b37-b9ab-b066f7456f1b-networking-console-plugin-cert podName:4529840b-b684-4b37-b9ab-b066f7456f1b nodeName:}" failed. No retries permitted until 2026-07-28 22:10:17.046634502 +0000 UTC m=+37.133892552 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/4529840b-b684-4b37-b9ab-b066f7456f1b-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-ncbpn" (UID: "4529840b-b684-4b37-b9ab-b066f7456f1b") : secret "networking-console-plugin-cert" not found Jul 28 22:10:16.046778 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:16.046731 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7e266916-99eb-4807-9b85-fabdeb9e1c28-service-ca-bundle podName:7e266916-99eb-4807-9b85-fabdeb9e1c28 nodeName:}" failed. No retries permitted until 2026-07-28 22:10:17.04669528 +0000 UTC m=+37.133953327 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7e266916-99eb-4807-9b85-fabdeb9e1c28-service-ca-bundle") pod "router-default-6747b7576c-fs2bd" (UID: "7e266916-99eb-4807-9b85-fabdeb9e1c28") : configmap references non-existent config key: service-ca.crt Jul 28 22:10:16.046778 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:16.046754 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2d6df48-f09b-4f10-b909-dedec2d5b3d0-samples-operator-tls podName:a2d6df48-f09b-4f10-b909-dedec2d5b3d0 nodeName:}" failed. No retries permitted until 2026-07-28 22:10:17.04674514 +0000 UTC m=+37.134003186 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a2d6df48-f09b-4f10-b909-dedec2d5b3d0-samples-operator-tls") pod "cluster-samples-operator-6748489656-8nnqd" (UID: "a2d6df48-f09b-4f10-b909-dedec2d5b3d0") : secret "samples-operator-tls" not found Jul 28 22:10:16.046778 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:16.046769 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eed632c4-ac4f-4f76-8a0e-a12bcf00835d-registry-tls podName:eed632c4-ac4f-4f76-8a0e-a12bcf00835d nodeName:}" failed. No retries permitted until 2026-07-28 22:10:17.046763807 +0000 UTC m=+37.134021854 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/eed632c4-ac4f-4f76-8a0e-a12bcf00835d-registry-tls") pod "image-registry-65465d7954-dqhtb" (UID: "eed632c4-ac4f-4f76-8a0e-a12bcf00835d") : secret "image-registry-tls" not found Jul 28 22:10:16.046983 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:16.046802 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eb1d24ec-acde-48ed-9004-aa372bc53057-metrics-tls\") pod \"dns-default-l6gf5\" (UID: \"eb1d24ec-acde-48ed-9004-aa372bc53057\") " pod="openshift-dns/dns-default-l6gf5" Jul 28 22:10:16.046983 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:16.046831 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/139cae0a-1747-4e84-b210-24cac8e94541-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-zn5fm\" (UID: \"139cae0a-1747-4e84-b210-24cac8e94541\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-zn5fm" Jul 28 22:10:16.046983 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:16.046849 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e266916-99eb-4807-9b85-fabdeb9e1c28-metrics-certs\") pod \"router-default-6747b7576c-fs2bd\" (UID: \"7e266916-99eb-4807-9b85-fabdeb9e1c28\") " pod="openshift-ingress/router-default-6747b7576c-fs2bd" Jul 28 22:10:16.046983 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:16.046947 2553 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 28 22:10:16.047129 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:16.046977 2553 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 28 22:10:16.047474 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:16.046929 2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 22:10:16.047524 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:16.047200 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/139cae0a-1747-4e84-b210-24cac8e94541-cluster-monitoring-operator-tls podName:139cae0a-1747-4e84-b210-24cac8e94541 nodeName:}" failed. No retries permitted until 2026-07-28 22:10:17.047177704 +0000 UTC m=+37.134435758 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/139cae0a-1747-4e84-b210-24cac8e94541-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-zn5fm" (UID: "139cae0a-1747-4e84-b210-24cac8e94541") : secret "cluster-monitoring-operator-tls" not found Jul 28 22:10:16.047564 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:16.047529 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e266916-99eb-4807-9b85-fabdeb9e1c28-metrics-certs podName:7e266916-99eb-4807-9b85-fabdeb9e1c28 nodeName:}" failed. No retries permitted until 2026-07-28 22:10:17.047508541 +0000 UTC m=+37.134766595 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7e266916-99eb-4807-9b85-fabdeb9e1c28-metrics-certs") pod "router-default-6747b7576c-fs2bd" (UID: "7e266916-99eb-4807-9b85-fabdeb9e1c28") : secret "router-metrics-certs-default" not found Jul 28 22:10:16.047564 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:16.047556 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb1d24ec-acde-48ed-9004-aa372bc53057-metrics-tls podName:eb1d24ec-acde-48ed-9004-aa372bc53057 nodeName:}" failed. No retries permitted until 2026-07-28 22:10:17.047546429 +0000 UTC m=+37.134804483 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/eb1d24ec-acde-48ed-9004-aa372bc53057-metrics-tls") pod "dns-default-l6gf5" (UID: "eb1d24ec-acde-48ed-9004-aa372bc53057") : secret "dns-default-metrics-tls" not found Jul 28 22:10:16.399585 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:16.399499 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-djzl8" Jul 28 22:10:16.399585 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:16.399551 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qz6gw" Jul 28 22:10:16.403133 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:16.401896 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 28 22:10:16.403133 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:16.401975 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7dq7d\"" Jul 28 22:10:16.716272 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:16.716063 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-q8wn6"] Jul 28 22:10:16.719618 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:16.719388 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-2dpdh"] Jul 28 22:10:16.722897 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:16.722832 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nzswh"] Jul 28 22:10:16.747964 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:16.747945 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-bgl6d"] Jul 28 22:10:16.750604 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:10:16.750580 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48d47548_1ab9_4983_8e36_9de23b54b911.slice/crio-efde8990187aee74d73517e757fed2776be6196e41f415d7ef8bd21e1da6be73 WatchSource:0}: Error finding container efde8990187aee74d73517e757fed2776be6196e41f415d7ef8bd21e1da6be73: Status 404 returned error can't find the container with id efde8990187aee74d73517e757fed2776be6196e41f415d7ef8bd21e1da6be73 Jul 28 22:10:16.773904 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:16.773874 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-bpt84"] Jul 28 22:10:16.775089 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:16.775000 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-7mp2z"] Jul 28 22:10:16.776547 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:16.775732 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-mldhp"] Jul 28 22:10:16.778500 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:10:16.778471 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1ced996_f136_4d3e_9c85_a5b7809b4517.slice/crio-ad3edf4f5af68e023a169132f10c7123654cd1a0f268ad3ea72e2f32b42fbd2c WatchSource:0}: Error finding container ad3edf4f5af68e023a169132f10c7123654cd1a0f268ad3ea72e2f32b42fbd2c: Status 404 returned error can't find the container with id ad3edf4f5af68e023a169132f10c7123654cd1a0f268ad3ea72e2f32b42fbd2c Jul 28 22:10:16.964144 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:16.964108 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7q942_c93df2e7-d79a-4a98-8556-a98868dfd93e/konnectivity-agent/0.log" limit=1024 Jul 28 22:10:17.062581 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:17.062542 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e266916-99eb-4807-9b85-fabdeb9e1c28-service-ca-bundle\") pod \"router-default-6747b7576c-fs2bd\" (UID: \"7e266916-99eb-4807-9b85-fabdeb9e1c28\") " pod="openshift-ingress/router-default-6747b7576c-fs2bd" Jul 28 22:10:17.062745 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:17.062609 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eb1d24ec-acde-48ed-9004-aa372bc53057-metrics-tls\") pod \"dns-default-l6gf5\" (UID: \"eb1d24ec-acde-48ed-9004-aa372bc53057\") " pod="openshift-dns/dns-default-l6gf5" Jul 28 22:10:17.062745 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:17.062655 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/139cae0a-1747-4e84-b210-24cac8e94541-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-zn5fm\" (UID: \"139cae0a-1747-4e84-b210-24cac8e94541\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-zn5fm" Jul 28 22:10:17.062745 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:17.062680 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e266916-99eb-4807-9b85-fabdeb9e1c28-metrics-certs\") pod \"router-default-6747b7576c-fs2bd\" (UID: \"7e266916-99eb-4807-9b85-fabdeb9e1c28\") " pod="openshift-ingress/router-default-6747b7576c-fs2bd" Jul 28 22:10:17.062897 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:17.062753 2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 22:10:17.062897 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:17.062779 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7e266916-99eb-4807-9b85-fabdeb9e1c28-service-ca-bundle podName:7e266916-99eb-4807-9b85-fabdeb9e1c28 nodeName:}" failed. No retries permitted until 2026-07-28 22:10:19.062750957 +0000 UTC m=+39.150009021 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7e266916-99eb-4807-9b85-fabdeb9e1c28-service-ca-bundle") pod "router-default-6747b7576c-fs2bd" (UID: "7e266916-99eb-4807-9b85-fabdeb9e1c28") : configmap references non-existent config key: service-ca.crt Jul 28 22:10:17.062897 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:17.062804 2553 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 28 22:10:17.062897 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:17.062814 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb1d24ec-acde-48ed-9004-aa372bc53057-metrics-tls podName:eb1d24ec-acde-48ed-9004-aa372bc53057 nodeName:}" failed. No retries permitted until 2026-07-28 22:10:19.062795805 +0000 UTC m=+39.150053874 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/eb1d24ec-acde-48ed-9004-aa372bc53057-metrics-tls") pod "dns-default-l6gf5" (UID: "eb1d24ec-acde-48ed-9004-aa372bc53057") : secret "dns-default-metrics-tls" not found Jul 28 22:10:17.062897 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:17.062833 2553 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 28 22:10:17.062897 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:17.062842 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/139cae0a-1747-4e84-b210-24cac8e94541-cluster-monitoring-operator-tls podName:139cae0a-1747-4e84-b210-24cac8e94541 nodeName:}" failed. No retries permitted until 2026-07-28 22:10:19.062831221 +0000 UTC m=+39.150089273 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/139cae0a-1747-4e84-b210-24cac8e94541-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-zn5fm" (UID: "139cae0a-1747-4e84-b210-24cac8e94541") : secret "cluster-monitoring-operator-tls" not found Jul 28 22:10:17.062897 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:17.062838 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4529840b-b684-4b37-b9ab-b066f7456f1b-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-ncbpn\" (UID: \"4529840b-b684-4b37-b9ab-b066f7456f1b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-ncbpn" Jul 28 22:10:17.062897 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:17.062869 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e266916-99eb-4807-9b85-fabdeb9e1c28-metrics-certs podName:7e266916-99eb-4807-9b85-fabdeb9e1c28 nodeName:}" failed. No retries permitted until 2026-07-28 22:10:19.062857973 +0000 UTC m=+39.150116046 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7e266916-99eb-4807-9b85-fabdeb9e1c28-metrics-certs") pod "router-default-6747b7576c-fs2bd" (UID: "7e266916-99eb-4807-9b85-fabdeb9e1c28") : secret "router-metrics-certs-default" not found Jul 28 22:10:17.063261 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:17.062901 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eed632c4-ac4f-4f76-8a0e-a12bcf00835d-registry-tls\") pod \"image-registry-65465d7954-dqhtb\" (UID: \"eed632c4-ac4f-4f76-8a0e-a12bcf00835d\") " pod="openshift-image-registry/image-registry-65465d7954-dqhtb" Jul 28 22:10:17.063261 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:17.062907 2553 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 28 22:10:17.063261 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:17.062944 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4529840b-b684-4b37-b9ab-b066f7456f1b-networking-console-plugin-cert podName:4529840b-b684-4b37-b9ab-b066f7456f1b nodeName:}" failed. No retries permitted until 2026-07-28 22:10:19.062937741 +0000 UTC m=+39.150195788 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/4529840b-b684-4b37-b9ab-b066f7456f1b-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-ncbpn" (UID: "4529840b-b684-4b37-b9ab-b066f7456f1b") : secret "networking-console-plugin-cert" not found Jul 28 22:10:17.063261 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:17.062977 2553 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 22:10:17.063261 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:17.062989 2553 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65465d7954-dqhtb: secret "image-registry-tls" not found Jul 28 22:10:17.063261 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:17.063019 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eed632c4-ac4f-4f76-8a0e-a12bcf00835d-registry-tls podName:eed632c4-ac4f-4f76-8a0e-a12bcf00835d nodeName:}" failed. No retries permitted until 2026-07-28 22:10:19.063009448 +0000 UTC m=+39.150267514 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/eed632c4-ac4f-4f76-8a0e-a12bcf00835d-registry-tls") pod "image-registry-65465d7954-dqhtb" (UID: "eed632c4-ac4f-4f76-8a0e-a12bcf00835d") : secret "image-registry-tls" not found Jul 28 22:10:17.063261 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:17.063047 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a2d6df48-f09b-4f10-b909-dedec2d5b3d0-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-8nnqd\" (UID: \"a2d6df48-f09b-4f10-b909-dedec2d5b3d0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-8nnqd" Jul 28 22:10:17.063261 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:17.063145 2553 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 28 22:10:17.063261 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:17.063181 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2d6df48-f09b-4f10-b909-dedec2d5b3d0-samples-operator-tls podName:a2d6df48-f09b-4f10-b909-dedec2d5b3d0 nodeName:}" failed. No retries permitted until 2026-07-28 22:10:19.063174075 +0000 UTC m=+39.150432121 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a2d6df48-f09b-4f10-b909-dedec2d5b3d0-samples-operator-tls") pod "cluster-samples-operator-6748489656-8nnqd" (UID: "a2d6df48-f09b-4f10-b909-dedec2d5b3d0") : secret "samples-operator-tls" not found Jul 28 22:10:17.064437 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:17.064402 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7q942_c93df2e7-d79a-4a98-8556-a98868dfd93e/konnectivity-agent/0.log" limit=1024 Jul 28 22:10:17.604815 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:17.601134 2553 generic.go:356] "Generic (PLEG): container finished" podID="327f1776-1215-4c6e-9fef-3e30c12b1b66" containerID="e1d01855e65dad422fdc0892b4113c3933ed0a9a0a5b6c5f9440d25a5eb5d351" exitCode=0 Jul 28 22:10:17.604815 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:17.601223 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zcxg2" event={"ID":"327f1776-1215-4c6e-9fef-3e30c12b1b66","Type":"ContainerDied","Data":"e1d01855e65dad422fdc0892b4113c3933ed0a9a0a5b6c5f9440d25a5eb5d351"} Jul 28 22:10:17.604815 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:17.604432 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-bgl6d" event={"ID":"48d47548-1ab9-4983-8e36-9de23b54b911","Type":"ContainerStarted","Data":"efde8990187aee74d73517e757fed2776be6196e41f415d7ef8bd21e1da6be73"} Jul 28 22:10:17.610833 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:17.610692 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-q8wn6" event={"ID":"8cea718f-40c0-48b7-943c-7c4178bd8d0d","Type":"ContainerStarted","Data":"2910efce8caff8e3d1d69af32a4b96be0606b8cf585749e903a352a42d154f23"} Jul 28 22:10:17.617402 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:17.617274 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-2dpdh" event={"ID":"b2f1e639-88f4-4c61-96ae-3a7b694e44e1","Type":"ContainerStarted","Data":"ab55e39f29053c7e45f50eada0ffdaccc48b4489d46ac88cf556e8cecfc1a646"} Jul 28 22:10:17.620602 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:17.620418 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-bpt84" event={"ID":"414a9396-e46b-433a-9976-95c5c0b1d897","Type":"ContainerStarted","Data":"607c95f34bad9db891b940fdb081d254784926c5855d2778e59b5427910db719"} Jul 28 22:10:17.628505 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:17.628479 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nzswh" event={"ID":"c1aff3e5-115b-4502-899a-b2137168973b","Type":"ContainerStarted","Data":"d0fc2e8ecdc64bbf23fc1b8a2bfd8b40d4b829fd44be9af968efecf7c6229f31"} Jul 28 22:10:17.633356 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:17.633333 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-7mp2z" event={"ID":"e1d395ef-80a9-4851-ad04-f6377dd12795","Type":"ContainerStarted","Data":"191d628e6002909eedde35e5fd723660f0f73cef31c0981c36cebf2589dd9868"} Jul 28 22:10:17.636662 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:17.636639 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-mldhp" event={"ID":"e1ced996-f136-4d3e-9c85-a5b7809b4517","Type":"ContainerStarted","Data":"ad3edf4f5af68e023a169132f10c7123654cd1a0f268ad3ea72e2f32b42fbd2c"} Jul 28 22:10:18.648724 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:18.648355 2553 generic.go:356] "Generic (PLEG): container finished" podID="327f1776-1215-4c6e-9fef-3e30c12b1b66" containerID="4432c1d98222db4e8a06f07a6b448eb8acf167d1ddaa6ab9eda067eea7cb779a" exitCode=0 Jul 28 22:10:18.648724 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:18.648504 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zcxg2" event={"ID":"327f1776-1215-4c6e-9fef-3e30c12b1b66","Type":"ContainerDied","Data":"4432c1d98222db4e8a06f07a6b448eb8acf167d1ddaa6ab9eda067eea7cb779a"} Jul 28 22:10:19.090308 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:19.090259 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4529840b-b684-4b37-b9ab-b066f7456f1b-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-ncbpn\" (UID: \"4529840b-b684-4b37-b9ab-b066f7456f1b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-ncbpn" Jul 28 22:10:19.090434 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:19.090313 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eed632c4-ac4f-4f76-8a0e-a12bcf00835d-registry-tls\") pod \"image-registry-65465d7954-dqhtb\" (UID: \"eed632c4-ac4f-4f76-8a0e-a12bcf00835d\") " pod="openshift-image-registry/image-registry-65465d7954-dqhtb" Jul 28 22:10:19.090434 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:19.090365 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a2d6df48-f09b-4f10-b909-dedec2d5b3d0-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-8nnqd\" (UID: \"a2d6df48-f09b-4f10-b909-dedec2d5b3d0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-8nnqd" Jul 28 22:10:19.090434 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:19.090399 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e266916-99eb-4807-9b85-fabdeb9e1c28-service-ca-bundle\") pod \"router-default-6747b7576c-fs2bd\" (UID: \"7e266916-99eb-4807-9b85-fabdeb9e1c28\") " pod="openshift-ingress/router-default-6747b7576c-fs2bd" Jul 28 22:10:19.090563 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:19.090447 2553 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 28 22:10:19.090563 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:19.090459 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eb1d24ec-acde-48ed-9004-aa372bc53057-metrics-tls\") pod \"dns-default-l6gf5\" (UID: \"eb1d24ec-acde-48ed-9004-aa372bc53057\") " pod="openshift-dns/dns-default-l6gf5" Jul 28 22:10:19.090563 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:19.090496 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/139cae0a-1747-4e84-b210-24cac8e94541-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-zn5fm\" (UID: \"139cae0a-1747-4e84-b210-24cac8e94541\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-zn5fm" Jul 28 22:10:19.090563 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:19.090520 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4529840b-b684-4b37-b9ab-b066f7456f1b-networking-console-plugin-cert podName:4529840b-b684-4b37-b9ab-b066f7456f1b nodeName:}" failed. No retries permitted until 2026-07-28 22:10:23.090498989 +0000 UTC m=+43.177757042 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/4529840b-b684-4b37-b9ab-b066f7456f1b-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-ncbpn" (UID: "4529840b-b684-4b37-b9ab-b066f7456f1b") : secret "networking-console-plugin-cert" not found Jul 28 22:10:19.090563 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:19.090559 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e266916-99eb-4807-9b85-fabdeb9e1c28-metrics-certs\") pod \"router-default-6747b7576c-fs2bd\" (UID: \"7e266916-99eb-4807-9b85-fabdeb9e1c28\") " pod="openshift-ingress/router-default-6747b7576c-fs2bd" Jul 28 22:10:19.090911 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:19.090584 2553 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 28 22:10:19.090911 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:19.090634 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/139cae0a-1747-4e84-b210-24cac8e94541-cluster-monitoring-operator-tls podName:139cae0a-1747-4e84-b210-24cac8e94541 nodeName:}" failed. No retries permitted until 2026-07-28 22:10:23.090618191 +0000 UTC m=+43.177876244 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/139cae0a-1747-4e84-b210-24cac8e94541-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-zn5fm" (UID: "139cae0a-1747-4e84-b210-24cac8e94541") : secret "cluster-monitoring-operator-tls" not found Jul 28 22:10:19.090911 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:19.090695 2553 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 22:10:19.090911 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:19.090723 2553 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65465d7954-dqhtb: secret "image-registry-tls" not found Jul 28 22:10:19.090911 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:19.090755 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eed632c4-ac4f-4f76-8a0e-a12bcf00835d-registry-tls podName:eed632c4-ac4f-4f76-8a0e-a12bcf00835d nodeName:}" failed. No retries permitted until 2026-07-28 22:10:23.090744738 +0000 UTC m=+43.178002788 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/eed632c4-ac4f-4f76-8a0e-a12bcf00835d-registry-tls") pod "image-registry-65465d7954-dqhtb" (UID: "eed632c4-ac4f-4f76-8a0e-a12bcf00835d") : secret "image-registry-tls" not found Jul 28 22:10:19.090911 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:19.090771 2553 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 28 22:10:19.090911 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:19.090807 2553 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 28 22:10:19.090911 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:19.090819 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e266916-99eb-4807-9b85-fabdeb9e1c28-metrics-certs podName:7e266916-99eb-4807-9b85-fabdeb9e1c28 nodeName:}" failed. No retries permitted until 2026-07-28 22:10:23.090804806 +0000 UTC m=+43.178062860 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7e266916-99eb-4807-9b85-fabdeb9e1c28-metrics-certs") pod "router-default-6747b7576c-fs2bd" (UID: "7e266916-99eb-4807-9b85-fabdeb9e1c28") : secret "router-metrics-certs-default" not found Jul 28 22:10:19.090911 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:19.090838 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2d6df48-f09b-4f10-b909-dedec2d5b3d0-samples-operator-tls podName:a2d6df48-f09b-4f10-b909-dedec2d5b3d0 nodeName:}" failed. No retries permitted until 2026-07-28 22:10:23.090828229 +0000 UTC m=+43.178086280 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a2d6df48-f09b-4f10-b909-dedec2d5b3d0-samples-operator-tls") pod "cluster-samples-operator-6748489656-8nnqd" (UID: "a2d6df48-f09b-4f10-b909-dedec2d5b3d0") : secret "samples-operator-tls" not found Jul 28 22:10:19.090911 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:19.090895 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7e266916-99eb-4807-9b85-fabdeb9e1c28-service-ca-bundle podName:7e266916-99eb-4807-9b85-fabdeb9e1c28 nodeName:}" failed. No retries permitted until 2026-07-28 22:10:23.090885312 +0000 UTC m=+43.178143360 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7e266916-99eb-4807-9b85-fabdeb9e1c28-service-ca-bundle") pod "router-default-6747b7576c-fs2bd" (UID: "7e266916-99eb-4807-9b85-fabdeb9e1c28") : configmap references non-existent config key: service-ca.crt Jul 28 22:10:19.092148 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:19.090945 2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 22:10:19.092148 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:19.090970 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb1d24ec-acde-48ed-9004-aa372bc53057-metrics-tls podName:eb1d24ec-acde-48ed-9004-aa372bc53057 nodeName:}" failed. No retries permitted until 2026-07-28 22:10:23.090962103 +0000 UTC m=+43.178220154 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/eb1d24ec-acde-48ed-9004-aa372bc53057-metrics-tls") pod "dns-default-l6gf5" (UID: "eb1d24ec-acde-48ed-9004-aa372bc53057") : secret "dns-default-metrics-tls" not found Jul 28 22:10:19.656436 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:19.656368 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zcxg2" event={"ID":"327f1776-1215-4c6e-9fef-3e30c12b1b66","Type":"ContainerStarted","Data":"500e9cd47b08e77a16301ba1229ffef6406e7a0563f427507086bca9d9a20174"} Jul 28 22:10:19.658326 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:19.658294 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-2dpdh" event={"ID":"b2f1e639-88f4-4c61-96ae-3a7b694e44e1","Type":"ContainerStarted","Data":"a4b1bc255e9b9874864b43effa276e1c6567ac30b0a6cb2fe6429c7eff909c30"} Jul 28 22:10:19.695664 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:19.695611 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-zcxg2" podStartSLOduration=10.190435597 podStartE2EDuration="39.695591206s" totalImagesPullingTime="29.505155609s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 22:09:40 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:10:19.679282031 +0000 UTC m=+39.766540099" watchObservedRunningTime="2026-07-28 22:10:19.695591206 +0000 UTC m=+39.782849275" Jul 28 22:10:19.695871 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:19.695820 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-2dpdh" podStartSLOduration=18.40811523 podStartE2EDuration="20.695811449s" totalImagesPullingTime="2.287696219s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:09:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:10:19.693245685 +0000 UTC m=+39.780503771" watchObservedRunningTime="2026-07-28 22:10:19.695811449 +0000 UTC m=+39.783069518" Jul 28 22:10:19.776759 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:19.776654 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7q942_c93df2e7-d79a-4a98-8556-a98868dfd93e/konnectivity-agent/0.log" limit=1024 Jul 28 22:10:19.833391 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:19.833347 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7q942_c93df2e7-d79a-4a98-8556-a98868dfd93e/konnectivity-agent/0.log" limit=1024 Jul 28 22:10:19.882497 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:19.882472 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7q942_c93df2e7-d79a-4a98-8556-a98868dfd93e/konnectivity-agent/0.log" limit=1024 Jul 28 22:10:23.128906 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:23.128803 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a2d6df48-f09b-4f10-b909-dedec2d5b3d0-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-8nnqd\" (UID: \"a2d6df48-f09b-4f10-b909-dedec2d5b3d0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-8nnqd" Jul 28 22:10:23.128906 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:23.128880 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e266916-99eb-4807-9b85-fabdeb9e1c28-service-ca-bundle\") pod \"router-default-6747b7576c-fs2bd\" (UID: \"7e266916-99eb-4807-9b85-fabdeb9e1c28\") " pod="openshift-ingress/router-default-6747b7576c-fs2bd" Jul 28 22:10:23.129578 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:23.128951 2553 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 28 22:10:23.129578 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:23.129010 2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 22:10:23.129578 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:23.129018 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2d6df48-f09b-4f10-b909-dedec2d5b3d0-samples-operator-tls podName:a2d6df48-f09b-4f10-b909-dedec2d5b3d0 nodeName:}" failed. No retries permitted until 2026-07-28 22:10:31.128999822 +0000 UTC m=+51.216257871 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a2d6df48-f09b-4f10-b909-dedec2d5b3d0-samples-operator-tls") pod "cluster-samples-operator-6748489656-8nnqd" (UID: "a2d6df48-f09b-4f10-b909-dedec2d5b3d0") : secret "samples-operator-tls" not found Jul 28 22:10:23.129578 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:23.128951 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eb1d24ec-acde-48ed-9004-aa372bc53057-metrics-tls\") pod \"dns-default-l6gf5\" (UID: \"eb1d24ec-acde-48ed-9004-aa372bc53057\") " pod="openshift-dns/dns-default-l6gf5" Jul 28 22:10:23.129578 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:23.129055 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7e266916-99eb-4807-9b85-fabdeb9e1c28-service-ca-bundle podName:7e266916-99eb-4807-9b85-fabdeb9e1c28 nodeName:}" failed. No retries permitted until 2026-07-28 22:10:31.129033943 +0000 UTC m=+51.216292013 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7e266916-99eb-4807-9b85-fabdeb9e1c28-service-ca-bundle") pod "router-default-6747b7576c-fs2bd" (UID: "7e266916-99eb-4807-9b85-fabdeb9e1c28") : configmap references non-existent config key: service-ca.crt Jul 28 22:10:23.129578 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:23.129097 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb1d24ec-acde-48ed-9004-aa372bc53057-metrics-tls podName:eb1d24ec-acde-48ed-9004-aa372bc53057 nodeName:}" failed. No retries permitted until 2026-07-28 22:10:31.12908459 +0000 UTC m=+51.216342673 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/eb1d24ec-acde-48ed-9004-aa372bc53057-metrics-tls") pod "dns-default-l6gf5" (UID: "eb1d24ec-acde-48ed-9004-aa372bc53057") : secret "dns-default-metrics-tls" not found Jul 28 22:10:23.129578 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:23.129139 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/139cae0a-1747-4e84-b210-24cac8e94541-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-zn5fm\" (UID: \"139cae0a-1747-4e84-b210-24cac8e94541\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-zn5fm" Jul 28 22:10:23.129578 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:23.129170 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e266916-99eb-4807-9b85-fabdeb9e1c28-metrics-certs\") pod \"router-default-6747b7576c-fs2bd\" (UID: \"7e266916-99eb-4807-9b85-fabdeb9e1c28\") " pod="openshift-ingress/router-default-6747b7576c-fs2bd" Jul 28 22:10:23.129578 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:23.129219 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4529840b-b684-4b37-b9ab-b066f7456f1b-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-ncbpn\" (UID: \"4529840b-b684-4b37-b9ab-b066f7456f1b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-ncbpn" Jul 28 22:10:23.129578 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:23.129251 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eed632c4-ac4f-4f76-8a0e-a12bcf00835d-registry-tls\") pod \"image-registry-65465d7954-dqhtb\" (UID: \"eed632c4-ac4f-4f76-8a0e-a12bcf00835d\") " pod="openshift-image-registry/image-registry-65465d7954-dqhtb" Jul 28 22:10:23.129578 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:23.129340 2553 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 22:10:23.129578 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:23.129350 2553 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65465d7954-dqhtb: secret "image-registry-tls" not found Jul 28 22:10:23.129578 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:23.129380 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eed632c4-ac4f-4f76-8a0e-a12bcf00835d-registry-tls podName:eed632c4-ac4f-4f76-8a0e-a12bcf00835d nodeName:}" failed. No retries permitted until 2026-07-28 22:10:31.12936915 +0000 UTC m=+51.216627219 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/eed632c4-ac4f-4f76-8a0e-a12bcf00835d-registry-tls") pod "image-registry-65465d7954-dqhtb" (UID: "eed632c4-ac4f-4f76-8a0e-a12bcf00835d") : secret "image-registry-tls" not found Jul 28 22:10:23.129578 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:23.129438 2553 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 28 22:10:23.129578 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:23.129487 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/139cae0a-1747-4e84-b210-24cac8e94541-cluster-monitoring-operator-tls podName:139cae0a-1747-4e84-b210-24cac8e94541 nodeName:}" failed. No retries permitted until 2026-07-28 22:10:31.129477147 +0000 UTC m=+51.216735194 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/139cae0a-1747-4e84-b210-24cac8e94541-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-zn5fm" (UID: "139cae0a-1747-4e84-b210-24cac8e94541") : secret "cluster-monitoring-operator-tls" not found Jul 28 22:10:23.129578 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:23.129552 2553 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 28 22:10:23.130316 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:23.129556 2553 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 28 22:10:23.130316 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:23.129593 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4529840b-b684-4b37-b9ab-b066f7456f1b-networking-console-plugin-cert podName:4529840b-b684-4b37-b9ab-b066f7456f1b nodeName:}" failed. No retries permitted until 2026-07-28 22:10:31.129586083 +0000 UTC m=+51.216844129 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/4529840b-b684-4b37-b9ab-b066f7456f1b-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-ncbpn" (UID: "4529840b-b684-4b37-b9ab-b066f7456f1b") : secret "networking-console-plugin-cert" not found Jul 28 22:10:23.130316 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:23.129630 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e266916-99eb-4807-9b85-fabdeb9e1c28-metrics-certs podName:7e266916-99eb-4807-9b85-fabdeb9e1c28 nodeName:}" failed. No retries permitted until 2026-07-28 22:10:31.129613787 +0000 UTC m=+51.216871853 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7e266916-99eb-4807-9b85-fabdeb9e1c28-metrics-certs") pod "router-default-6747b7576c-fs2bd" (UID: "7e266916-99eb-4807-9b85-fabdeb9e1c28") : secret "router-metrics-certs-default" not found Jul 28 22:10:24.674632 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:24.674600 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-7mp2z" event={"ID":"e1d395ef-80a9-4851-ad04-f6377dd12795","Type":"ContainerStarted","Data":"30698e93ae7f59980a2eda0481fdf9d4135e16716211abee531f547bd74fbc88"} Jul 28 22:10:24.702509 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:24.702458 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-7mp2z" podStartSLOduration=17.96759728 podStartE2EDuration="25.702437574s" totalImagesPullingTime="7.734840294s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:09:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:10:24.701859247 +0000 UTC m=+44.789117317" watchObservedRunningTime="2026-07-28 22:10:24.702437574 +0000 UTC m=+44.789695638" Jul 28 22:10:25.680748 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:25.680691 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nzswh" event={"ID":"c1aff3e5-115b-4502-899a-b2137168973b","Type":"ContainerStarted","Data":"1f27a3dc4bd7d2437c187d6243ac6dedff608beb427b95077bca8e73e249e302"} Jul 28 22:10:25.682783 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:25.682754 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mldhp_e1ced996-f136-4d3e-9c85-a5b7809b4517/console-operator/0.log" Jul 28 22:10:25.682898 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:25.682799 2553 generic.go:356] "Generic (PLEG): container finished" podID="e1ced996-f136-4d3e-9c85-a5b7809b4517" containerID="b6f38d89af24a2ddbb0875bfb047d93e302be97ff53e2183826ea0f044cf2bad" exitCode=255 Jul 28 22:10:25.682969 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:25.682886 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-mldhp" event={"ID":"e1ced996-f136-4d3e-9c85-a5b7809b4517","Type":"ContainerDied","Data":"b6f38d89af24a2ddbb0875bfb047d93e302be97ff53e2183826ea0f044cf2bad"} Jul 28 22:10:25.683062 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:25.683041 2553 scope.go:117] "RemoveContainer" containerID="b6f38d89af24a2ddbb0875bfb047d93e302be97ff53e2183826ea0f044cf2bad" Jul 28 22:10:25.686801 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:25.686742 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-bgl6d" event={"ID":"48d47548-1ab9-4983-8e36-9de23b54b911","Type":"ContainerStarted","Data":"0d337f9faff9bab14e68abe56c301b7d62c8c7699f58be343df5b00fc06884ec"} Jul 28 22:10:25.688580 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:25.688556 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-q8wn6" event={"ID":"8cea718f-40c0-48b7-943c-7c4178bd8d0d","Type":"ContainerStarted","Data":"660efcd8022960904dcf8e6d7a83aabc17af2b0cbb0298d548051592aba8ef47"} Jul 28 22:10:25.690144 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:25.690119 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-bpt84" event={"ID":"414a9396-e46b-433a-9976-95c5c0b1d897","Type":"ContainerStarted","Data":"0b9d4645e52ed06281a88ee6132a90771294621d17f602e623280ba38a5f0920"} Jul 28 22:10:25.697767 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:25.697720 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nzswh" podStartSLOduration=18.917318569 podStartE2EDuration="26.697683936s" totalImagesPullingTime="7.780365367s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:09:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:10:25.696917798 +0000 UTC m=+45.784175894" watchObservedRunningTime="2026-07-28 22:10:25.697683936 +0000 UTC m=+45.784942006" Jul 28 22:10:25.745289 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:25.745237 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-q8wn6" podStartSLOduration=2.961915864 podStartE2EDuration="10.745218578s" totalImagesPullingTime="7.783302714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:10:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:10:25.744684387 +0000 UTC m=+45.831942456" watchObservedRunningTime="2026-07-28 22:10:25.745218578 +0000 UTC m=+45.832476647" Jul 28 22:10:25.746042 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:25.746000 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-bpt84" podStartSLOduration=19.012229606 podStartE2EDuration="26.745988038s" totalImagesPullingTime="7.733758432s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:09:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:10:25.731340041 +0000 UTC m=+45.818598111" watchObservedRunningTime="2026-07-28 22:10:25.745988038 +0000 UTC m=+45.833246106" Jul 28 22:10:25.761080 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:25.761038 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-bgl6d" podStartSLOduration=18.985460939 podStartE2EDuration="26.760995248s" totalImagesPullingTime="7.775534309s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:09:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:10:25.760214673 +0000 UTC m=+45.847472743" watchObservedRunningTime="2026-07-28 22:10:25.760995248 +0000 UTC m=+45.848253321" Jul 28 22:10:26.694646 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:26.694596 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mldhp_e1ced996-f136-4d3e-9c85-a5b7809b4517/console-operator/1.log" Jul 28 22:10:26.695099 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:26.695062 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mldhp_e1ced996-f136-4d3e-9c85-a5b7809b4517/console-operator/0.log" Jul 28 22:10:26.695157 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:26.695101 2553 generic.go:356] "Generic (PLEG): container finished" podID="e1ced996-f136-4d3e-9c85-a5b7809b4517" containerID="6cbbf9649882f41080c11db3f7051f60a03a344840afbabb6ea6b9bc0dab4e9d" exitCode=255 Jul 28 22:10:26.695256 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:26.695221 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-mldhp" event={"ID":"e1ced996-f136-4d3e-9c85-a5b7809b4517","Type":"ContainerDied","Data":"6cbbf9649882f41080c11db3f7051f60a03a344840afbabb6ea6b9bc0dab4e9d"} Jul 28 22:10:26.695331 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:26.695319 2553 scope.go:117] "RemoveContainer" containerID="b6f38d89af24a2ddbb0875bfb047d93e302be97ff53e2183826ea0f044cf2bad" Jul 28 22:10:26.695594 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:26.695573 2553 scope.go:117] "RemoveContainer" containerID="6cbbf9649882f41080c11db3f7051f60a03a344840afbabb6ea6b9bc0dab4e9d" Jul 28 22:10:26.695842 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:26.695797 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-mldhp_openshift-console-operator(e1ced996-f136-4d3e-9c85-a5b7809b4517)\"" pod="openshift-console-operator/console-operator-584dcb6b54-mldhp" podUID="e1ced996-f136-4d3e-9c85-a5b7809b4517" Jul 28 22:10:26.744429 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:26.744382 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-wx69k"] Jul 28 22:10:26.772814 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:26.772783 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wx69k" Jul 28 22:10:26.776205 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:26.776183 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 28 22:10:26.776316 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:26.776185 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-r6q89\"" Jul 28 22:10:26.776316 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:26.776190 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 28 22:10:26.780502 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:26.780458 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wx69k"] Jul 28 22:10:26.864182 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:26.864154 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c004d0a-52da-428b-b2f3-aefe98d15909-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wx69k\" (UID: \"4c004d0a-52da-428b-b2f3-aefe98d15909\") " pod="openshift-insights/insights-runtime-extractor-wx69k" Jul 28 22:10:26.864300 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:26.864198 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4c004d0a-52da-428b-b2f3-aefe98d15909-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wx69k\" (UID: \"4c004d0a-52da-428b-b2f3-aefe98d15909\") " pod="openshift-insights/insights-runtime-extractor-wx69k" Jul 28 22:10:26.864300 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:26.864290 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnn6x\" (UniqueName: \"kubernetes.io/projected/4c004d0a-52da-428b-b2f3-aefe98d15909-kube-api-access-tnn6x\") pod \"insights-runtime-extractor-wx69k\" (UID: \"4c004d0a-52da-428b-b2f3-aefe98d15909\") " pod="openshift-insights/insights-runtime-extractor-wx69k" Jul 28 22:10:26.864379 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:26.864351 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4c004d0a-52da-428b-b2f3-aefe98d15909-crio-socket\") pod \"insights-runtime-extractor-wx69k\" (UID: \"4c004d0a-52da-428b-b2f3-aefe98d15909\") " pod="openshift-insights/insights-runtime-extractor-wx69k" Jul 28 22:10:26.864379 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:26.864372 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4c004d0a-52da-428b-b2f3-aefe98d15909-data-volume\") pod \"insights-runtime-extractor-wx69k\" (UID: \"4c004d0a-52da-428b-b2f3-aefe98d15909\") " pod="openshift-insights/insights-runtime-extractor-wx69k" Jul 28 22:10:26.965761 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:26.965734 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c004d0a-52da-428b-b2f3-aefe98d15909-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wx69k\" (UID: \"4c004d0a-52da-428b-b2f3-aefe98d15909\") " pod="openshift-insights/insights-runtime-extractor-wx69k" Jul 28 22:10:26.965927 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:26.965777 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4c004d0a-52da-428b-b2f3-aefe98d15909-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wx69k\" (UID: \"4c004d0a-52da-428b-b2f3-aefe98d15909\") " pod="openshift-insights/insights-runtime-extractor-wx69k" Jul 28 22:10:26.965927 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:26.965871 2553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 22:10:26.965927 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:26.965899 2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 22:10:26.965927 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:26.965929 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4c004d0a-52da-428b-b2f3-aefe98d15909-kube-rbac-proxy-cm podName:4c004d0a-52da-428b-b2f3-aefe98d15909 nodeName:}" failed. No retries permitted until 2026-07-28 22:10:27.465915432 +0000 UTC m=+47.553173479 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4c004d0a-52da-428b-b2f3-aefe98d15909-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wx69k" (UID: "4c004d0a-52da-428b-b2f3-aefe98d15909") : configmap "kube-rbac-proxy" not found Jul 28 22:10:26.966142 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:26.965953 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnn6x\" (UniqueName: \"kubernetes.io/projected/4c004d0a-52da-428b-b2f3-aefe98d15909-kube-api-access-tnn6x\") pod \"insights-runtime-extractor-wx69k\" (UID: \"4c004d0a-52da-428b-b2f3-aefe98d15909\") " pod="openshift-insights/insights-runtime-extractor-wx69k" Jul 28 22:10:26.966142 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:26.965963 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c004d0a-52da-428b-b2f3-aefe98d15909-insights-runtime-extractor-tls podName:4c004d0a-52da-428b-b2f3-aefe98d15909 nodeName:}" failed. No retries permitted until 2026-07-28 22:10:27.465943182 +0000 UTC m=+47.553201256 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4c004d0a-52da-428b-b2f3-aefe98d15909-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wx69k" (UID: "4c004d0a-52da-428b-b2f3-aefe98d15909") : secret "insights-runtime-extractor-tls" not found Jul 28 22:10:26.966142 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:26.966050 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4c004d0a-52da-428b-b2f3-aefe98d15909-crio-socket\") pod \"insights-runtime-extractor-wx69k\" (UID: \"4c004d0a-52da-428b-b2f3-aefe98d15909\") " pod="openshift-insights/insights-runtime-extractor-wx69k" Jul 28 22:10:26.966142 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:26.966075 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4c004d0a-52da-428b-b2f3-aefe98d15909-data-volume\") pod \"insights-runtime-extractor-wx69k\" (UID: \"4c004d0a-52da-428b-b2f3-aefe98d15909\") " pod="openshift-insights/insights-runtime-extractor-wx69k" Jul 28 22:10:26.966295 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:26.966182 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4c004d0a-52da-428b-b2f3-aefe98d15909-crio-socket\") pod \"insights-runtime-extractor-wx69k\" (UID: \"4c004d0a-52da-428b-b2f3-aefe98d15909\") " pod="openshift-insights/insights-runtime-extractor-wx69k" Jul 28 22:10:26.966398 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:26.966383 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4c004d0a-52da-428b-b2f3-aefe98d15909-data-volume\") pod \"insights-runtime-extractor-wx69k\" (UID: \"4c004d0a-52da-428b-b2f3-aefe98d15909\") " pod="openshift-insights/insights-runtime-extractor-wx69k" Jul 28 22:10:26.970321 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:26.970291 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-x7ks4"] Jul 28 22:10:26.984082 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:26.984059 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnn6x\" (UniqueName: \"kubernetes.io/projected/4c004d0a-52da-428b-b2f3-aefe98d15909-kube-api-access-tnn6x\") pod \"insights-runtime-extractor-wx69k\" (UID: \"4c004d0a-52da-428b-b2f3-aefe98d15909\") " pod="openshift-insights/insights-runtime-extractor-wx69k" Jul 28 22:10:26.996599 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:26.996575 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-x7ks4"] Jul 28 22:10:26.996693 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:26.996658 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-x7ks4" Jul 28 22:10:26.999061 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:26.999042 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-98cfd\"" Jul 28 22:10:26.999061 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:26.999057 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 28 22:10:26.999225 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:26.999091 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 28 22:10:27.067163 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:27.067137 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bvkv\" (UniqueName: \"kubernetes.io/projected/fc8b5f1d-2ee7-4775-8ee8-cc1175781678-kube-api-access-5bvkv\") pod \"migrator-6865c9967-x7ks4\" (UID: \"fc8b5f1d-2ee7-4775-8ee8-cc1175781678\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-x7ks4" Jul 28 22:10:27.168248 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:27.168219 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bvkv\" (UniqueName: \"kubernetes.io/projected/fc8b5f1d-2ee7-4775-8ee8-cc1175781678-kube-api-access-5bvkv\") pod \"migrator-6865c9967-x7ks4\" (UID: \"fc8b5f1d-2ee7-4775-8ee8-cc1175781678\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-x7ks4" Jul 28 22:10:27.175922 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:27.175894 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bvkv\" (UniqueName: \"kubernetes.io/projected/fc8b5f1d-2ee7-4775-8ee8-cc1175781678-kube-api-access-5bvkv\") pod \"migrator-6865c9967-x7ks4\" (UID: \"fc8b5f1d-2ee7-4775-8ee8-cc1175781678\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-x7ks4" Jul 28 22:10:27.305633 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:27.305566 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-x7ks4" Jul 28 22:10:27.450128 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:27.450099 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-x7ks4"] Jul 28 22:10:27.452890 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:10:27.452861 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc8b5f1d_2ee7_4775_8ee8_cc1175781678.slice/crio-d56bf2ee126cc8ab33ecff85cf254279228e958de367dc51e4233c6f36157e5e WatchSource:0}: Error finding container d56bf2ee126cc8ab33ecff85cf254279228e958de367dc51e4233c6f36157e5e: Status 404 returned error can't find the container with id d56bf2ee126cc8ab33ecff85cf254279228e958de367dc51e4233c6f36157e5e Jul 28 22:10:27.471398 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:27.471372 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c004d0a-52da-428b-b2f3-aefe98d15909-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wx69k\" (UID: \"4c004d0a-52da-428b-b2f3-aefe98d15909\") " pod="openshift-insights/insights-runtime-extractor-wx69k" Jul 28 22:10:27.471489 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:27.471412 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4c004d0a-52da-428b-b2f3-aefe98d15909-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wx69k\" (UID: \"4c004d0a-52da-428b-b2f3-aefe98d15909\") " pod="openshift-insights/insights-runtime-extractor-wx69k" Jul 28 22:10:27.471560 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:27.471541 2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 22:10:27.471610 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:27.471547 2553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 22:10:27.471610 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:27.471603 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c004d0a-52da-428b-b2f3-aefe98d15909-insights-runtime-extractor-tls podName:4c004d0a-52da-428b-b2f3-aefe98d15909 nodeName:}" failed. No retries permitted until 2026-07-28 22:10:28.471588419 +0000 UTC m=+48.558846470 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4c004d0a-52da-428b-b2f3-aefe98d15909-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wx69k" (UID: "4c004d0a-52da-428b-b2f3-aefe98d15909") : secret "insights-runtime-extractor-tls" not found Jul 28 22:10:27.471690 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:27.471618 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4c004d0a-52da-428b-b2f3-aefe98d15909-kube-rbac-proxy-cm podName:4c004d0a-52da-428b-b2f3-aefe98d15909 nodeName:}" failed. No retries permitted until 2026-07-28 22:10:28.471610575 +0000 UTC m=+48.558868621 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4c004d0a-52da-428b-b2f3-aefe98d15909-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wx69k" (UID: "4c004d0a-52da-428b-b2f3-aefe98d15909") : configmap "kube-rbac-proxy" not found Jul 28 22:10:27.699872 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:27.699791 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mldhp_e1ced996-f136-4d3e-9c85-a5b7809b4517/console-operator/1.log" Jul 28 22:10:27.700262 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:27.700142 2553 scope.go:117] "RemoveContainer" containerID="6cbbf9649882f41080c11db3f7051f60a03a344840afbabb6ea6b9bc0dab4e9d" Jul 28 22:10:27.700324 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:27.700291 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-mldhp_openshift-console-operator(e1ced996-f136-4d3e-9c85-a5b7809b4517)\"" pod="openshift-console-operator/console-operator-584dcb6b54-mldhp" podUID="e1ced996-f136-4d3e-9c85-a5b7809b4517" Jul 28 22:10:27.701172 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:27.701149 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-x7ks4" event={"ID":"fc8b5f1d-2ee7-4775-8ee8-cc1175781678","Type":"ContainerStarted","Data":"d56bf2ee126cc8ab33ecff85cf254279228e958de367dc51e4233c6f36157e5e"} Jul 28 22:10:28.124721 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:28.124689 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-w9bnn_0f57ff96-5ed8-4a73-9dfd-7e5d45bd163c/dns-node-resolver/0.log" Jul 28 22:10:28.481790 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:28.481753 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c004d0a-52da-428b-b2f3-aefe98d15909-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wx69k\" (UID: \"4c004d0a-52da-428b-b2f3-aefe98d15909\") " pod="openshift-insights/insights-runtime-extractor-wx69k" Jul 28 22:10:28.481989 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:28.481807 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4c004d0a-52da-428b-b2f3-aefe98d15909-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wx69k\" (UID: \"4c004d0a-52da-428b-b2f3-aefe98d15909\") " pod="openshift-insights/insights-runtime-extractor-wx69k" Jul 28 22:10:28.481989 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:28.481912 2553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 22:10:28.481989 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:28.481947 2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 22:10:28.481989 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:28.481983 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4c004d0a-52da-428b-b2f3-aefe98d15909-kube-rbac-proxy-cm podName:4c004d0a-52da-428b-b2f3-aefe98d15909 nodeName:}" failed. No retries permitted until 2026-07-28 22:10:30.481966308 +0000 UTC m=+50.569224355 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4c004d0a-52da-428b-b2f3-aefe98d15909-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wx69k" (UID: "4c004d0a-52da-428b-b2f3-aefe98d15909") : configmap "kube-rbac-proxy" not found Jul 28 22:10:28.482182 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:28.482014 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c004d0a-52da-428b-b2f3-aefe98d15909-insights-runtime-extractor-tls podName:4c004d0a-52da-428b-b2f3-aefe98d15909 nodeName:}" failed. No retries permitted until 2026-07-28 22:10:30.481994497 +0000 UTC m=+50.569252549 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4c004d0a-52da-428b-b2f3-aefe98d15909-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wx69k" (UID: "4c004d0a-52da-428b-b2f3-aefe98d15909") : secret "insights-runtime-extractor-tls" not found Jul 28 22:10:28.924205 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:28.924125 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bs64q_54dc150c-a1f7-4df4-9adf-03077bf59073/node-ca/0.log" Jul 28 22:10:29.527885 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:29.527859 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-q8wn6_8cea718f-40c0-48b7-943c-7c4178bd8d0d/serve-healthcheck-canary/0.log" Jul 28 22:10:29.710357 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:29.710320 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-x7ks4" event={"ID":"fc8b5f1d-2ee7-4775-8ee8-cc1175781678","Type":"ContainerStarted","Data":"d38b8a6622a744b7ca2db781354a0e933ad32b1fef4f9db46ea9696d390c7989"} Jul 28 22:10:29.710520 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:29.710422 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-x7ks4" event={"ID":"fc8b5f1d-2ee7-4775-8ee8-cc1175781678","Type":"ContainerStarted","Data":"baeefbaac155367026e406f17fecf6c149f8fba25c6064a70b7aa286c0848ab6"} Jul 28 22:10:29.730140 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:29.730095 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-x7ks4" podStartSLOduration=2.244077208 podStartE2EDuration="3.730078676s" totalImagesPullingTime="1.486001468s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:10:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:10:29.729009367 +0000 UTC m=+49.816267436" watchObservedRunningTime="2026-07-28 22:10:29.730078676 +0000 UTC m=+49.817336744" Jul 28 22:10:30.497149 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:30.497116 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c004d0a-52da-428b-b2f3-aefe98d15909-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wx69k\" (UID: \"4c004d0a-52da-428b-b2f3-aefe98d15909\") " pod="openshift-insights/insights-runtime-extractor-wx69k" Jul 28 22:10:30.497565 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:30.497162 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4c004d0a-52da-428b-b2f3-aefe98d15909-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wx69k\" (UID: \"4c004d0a-52da-428b-b2f3-aefe98d15909\") " pod="openshift-insights/insights-runtime-extractor-wx69k" Jul 28 22:10:30.497565 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:30.497228 2553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 22:10:30.497565 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:30.497263 2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 22:10:30.497565 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:30.497277 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4c004d0a-52da-428b-b2f3-aefe98d15909-kube-rbac-proxy-cm podName:4c004d0a-52da-428b-b2f3-aefe98d15909 nodeName:}" failed. No retries permitted until 2026-07-28 22:10:34.497263512 +0000 UTC m=+54.584521559 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4c004d0a-52da-428b-b2f3-aefe98d15909-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wx69k" (UID: "4c004d0a-52da-428b-b2f3-aefe98d15909") : configmap "kube-rbac-proxy" not found Jul 28 22:10:30.497565 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:30.497428 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c004d0a-52da-428b-b2f3-aefe98d15909-insights-runtime-extractor-tls podName:4c004d0a-52da-428b-b2f3-aefe98d15909 nodeName:}" failed. No retries permitted until 2026-07-28 22:10:34.497410302 +0000 UTC m=+54.584668351 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4c004d0a-52da-428b-b2f3-aefe98d15909-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wx69k" (UID: "4c004d0a-52da-428b-b2f3-aefe98d15909") : secret "insights-runtime-extractor-tls" not found Jul 28 22:10:31.203631 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:31.203590 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eed632c4-ac4f-4f76-8a0e-a12bcf00835d-registry-tls\") pod \"image-registry-65465d7954-dqhtb\" (UID: \"eed632c4-ac4f-4f76-8a0e-a12bcf00835d\") " pod="openshift-image-registry/image-registry-65465d7954-dqhtb" Jul 28 22:10:31.203876 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:31.203648 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a2d6df48-f09b-4f10-b909-dedec2d5b3d0-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-8nnqd\" (UID: \"a2d6df48-f09b-4f10-b909-dedec2d5b3d0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-8nnqd" Jul 28 22:10:31.203876 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:31.203686 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e266916-99eb-4807-9b85-fabdeb9e1c28-service-ca-bundle\") pod \"router-default-6747b7576c-fs2bd\" (UID: \"7e266916-99eb-4807-9b85-fabdeb9e1c28\") " pod="openshift-ingress/router-default-6747b7576c-fs2bd" Jul 28 22:10:31.203876 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:31.203747 2553 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 22:10:31.203876 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:31.203755 2553 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 28 22:10:31.203876 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:31.203766 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eb1d24ec-acde-48ed-9004-aa372bc53057-metrics-tls\") pod \"dns-default-l6gf5\" (UID: \"eb1d24ec-acde-48ed-9004-aa372bc53057\") " pod="openshift-dns/dns-default-l6gf5" Jul 28 22:10:31.203876 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:31.203828 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7e266916-99eb-4807-9b85-fabdeb9e1c28-service-ca-bundle podName:7e266916-99eb-4807-9b85-fabdeb9e1c28 nodeName:}" failed. No retries permitted until 2026-07-28 22:10:47.203813094 +0000 UTC m=+67.291071141 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7e266916-99eb-4807-9b85-fabdeb9e1c28-service-ca-bundle") pod "router-default-6747b7576c-fs2bd" (UID: "7e266916-99eb-4807-9b85-fabdeb9e1c28") : configmap references non-existent config key: service-ca.crt Jul 28 22:10:31.203876 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:31.203837 2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 22:10:31.203876 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:31.203817 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/139cae0a-1747-4e84-b210-24cac8e94541-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-zn5fm\" (UID: \"139cae0a-1747-4e84-b210-24cac8e94541\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-zn5fm" Jul 28 22:10:31.203876 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:31.203853 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2d6df48-f09b-4f10-b909-dedec2d5b3d0-samples-operator-tls podName:a2d6df48-f09b-4f10-b909-dedec2d5b3d0 nodeName:}" failed. No retries permitted until 2026-07-28 22:10:47.203846495 +0000 UTC m=+67.291104541 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a2d6df48-f09b-4f10-b909-dedec2d5b3d0-samples-operator-tls") pod "cluster-samples-operator-6748489656-8nnqd" (UID: "a2d6df48-f09b-4f10-b909-dedec2d5b3d0") : secret "samples-operator-tls" not found Jul 28 22:10:31.203876 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:31.203886 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb1d24ec-acde-48ed-9004-aa372bc53057-metrics-tls podName:eb1d24ec-acde-48ed-9004-aa372bc53057 nodeName:}" failed. No retries permitted until 2026-07-28 22:10:47.203872532 +0000 UTC m=+67.291130578 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/eb1d24ec-acde-48ed-9004-aa372bc53057-metrics-tls") pod "dns-default-l6gf5" (UID: "eb1d24ec-acde-48ed-9004-aa372bc53057") : secret "dns-default-metrics-tls" not found Jul 28 22:10:31.204242 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:31.203895 2553 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 28 22:10:31.204242 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:31.203767 2553 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65465d7954-dqhtb: secret "image-registry-tls" not found Jul 28 22:10:31.204242 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:31.203907 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e266916-99eb-4807-9b85-fabdeb9e1c28-metrics-certs\") pod \"router-default-6747b7576c-fs2bd\" (UID: \"7e266916-99eb-4807-9b85-fabdeb9e1c28\") " pod="openshift-ingress/router-default-6747b7576c-fs2bd" Jul 28 22:10:31.204242 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:31.203932 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eed632c4-ac4f-4f76-8a0e-a12bcf00835d-registry-tls podName:eed632c4-ac4f-4f76-8a0e-a12bcf00835d nodeName:}" failed. No retries permitted until 2026-07-28 22:10:47.203922062 +0000 UTC m=+67.291180141 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/eed632c4-ac4f-4f76-8a0e-a12bcf00835d-registry-tls") pod "image-registry-65465d7954-dqhtb" (UID: "eed632c4-ac4f-4f76-8a0e-a12bcf00835d") : secret "image-registry-tls" not found Jul 28 22:10:31.204242 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:31.203983 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4529840b-b684-4b37-b9ab-b066f7456f1b-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-ncbpn\" (UID: \"4529840b-b684-4b37-b9ab-b066f7456f1b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-ncbpn" Jul 28 22:10:31.204242 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:31.203995 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/139cae0a-1747-4e84-b210-24cac8e94541-cluster-monitoring-operator-tls podName:139cae0a-1747-4e84-b210-24cac8e94541 nodeName:}" failed. No retries permitted until 2026-07-28 22:10:47.203984706 +0000 UTC m=+67.291242754 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/139cae0a-1747-4e84-b210-24cac8e94541-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-zn5fm" (UID: "139cae0a-1747-4e84-b210-24cac8e94541") : secret "cluster-monitoring-operator-tls" not found Jul 28 22:10:31.204242 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:31.203999 2553 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 28 22:10:31.204242 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:31.204037 2553 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 28 22:10:31.204242 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:31.204055 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e266916-99eb-4807-9b85-fabdeb9e1c28-metrics-certs podName:7e266916-99eb-4807-9b85-fabdeb9e1c28 nodeName:}" failed. No retries permitted until 2026-07-28 22:10:47.204040663 +0000 UTC m=+67.291298724 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7e266916-99eb-4807-9b85-fabdeb9e1c28-metrics-certs") pod "router-default-6747b7576c-fs2bd" (UID: "7e266916-99eb-4807-9b85-fabdeb9e1c28") : secret "router-metrics-certs-default" not found Jul 28 22:10:31.204242 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:31.204077 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4529840b-b684-4b37-b9ab-b066f7456f1b-networking-console-plugin-cert podName:4529840b-b684-4b37-b9ab-b066f7456f1b nodeName:}" failed. No retries permitted until 2026-07-28 22:10:47.204066332 +0000 UTC m=+67.291324380 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/4529840b-b684-4b37-b9ab-b066f7456f1b-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-ncbpn" (UID: "4529840b-b684-4b37-b9ab-b066f7456f1b") : secret "networking-console-plugin-cert" not found Jul 28 22:10:34.533491 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:34.533448 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c004d0a-52da-428b-b2f3-aefe98d15909-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wx69k\" (UID: \"4c004d0a-52da-428b-b2f3-aefe98d15909\") " pod="openshift-insights/insights-runtime-extractor-wx69k" Jul 28 22:10:34.534046 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:34.533506 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4c004d0a-52da-428b-b2f3-aefe98d15909-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wx69k\" (UID: \"4c004d0a-52da-428b-b2f3-aefe98d15909\") " pod="openshift-insights/insights-runtime-extractor-wx69k" Jul 28 22:10:34.534046 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:34.533616 2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 22:10:34.534046 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:34.533678 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c004d0a-52da-428b-b2f3-aefe98d15909-insights-runtime-extractor-tls podName:4c004d0a-52da-428b-b2f3-aefe98d15909 nodeName:}" failed. No retries permitted until 2026-07-28 22:10:42.533663752 +0000 UTC m=+62.620921802 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4c004d0a-52da-428b-b2f3-aefe98d15909-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wx69k" (UID: "4c004d0a-52da-428b-b2f3-aefe98d15909") : secret "insights-runtime-extractor-tls" not found Jul 28 22:10:34.534157 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:34.534124 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4c004d0a-52da-428b-b2f3-aefe98d15909-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wx69k\" (UID: \"4c004d0a-52da-428b-b2f3-aefe98d15909\") " pod="openshift-insights/insights-runtime-extractor-wx69k" Jul 28 22:10:35.660406 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:35.660362 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-mldhp" Jul 28 22:10:35.660831 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:35.660432 2553 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-mldhp" Jul 28 22:10:35.660831 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:35.660781 2553 scope.go:117] "RemoveContainer" containerID="6cbbf9649882f41080c11db3f7051f60a03a344840afbabb6ea6b9bc0dab4e9d" Jul 28 22:10:35.660936 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:10:35.660917 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-mldhp_openshift-console-operator(e1ced996-f136-4d3e-9c85-a5b7809b4517)\"" pod="openshift-console-operator/console-operator-584dcb6b54-mldhp" podUID="e1ced996-f136-4d3e-9c85-a5b7809b4517" Jul 28 22:10:41.570500 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:41.570476 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vzbxk" Jul 28 22:10:42.603578 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:42.603541 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c004d0a-52da-428b-b2f3-aefe98d15909-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wx69k\" (UID: \"4c004d0a-52da-428b-b2f3-aefe98d15909\") " pod="openshift-insights/insights-runtime-extractor-wx69k" Jul 28 22:10:42.605971 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:42.605951 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c004d0a-52da-428b-b2f3-aefe98d15909-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wx69k\" (UID: \"4c004d0a-52da-428b-b2f3-aefe98d15909\") " pod="openshift-insights/insights-runtime-extractor-wx69k" Jul 28 22:10:42.685053 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:42.685021 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-r6q89\"" Jul 28 22:10:42.693049 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:42.693027 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wx69k" Jul 28 22:10:42.821741 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:42.821721 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wx69k"] Jul 28 22:10:42.824240 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:10:42.824215 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c004d0a_52da_428b_b2f3_aefe98d15909.slice/crio-c857a961857c83087c1854f8725aadb64b6e320bb863a3a4b3a873e33d6c7613 WatchSource:0}: Error finding container c857a961857c83087c1854f8725aadb64b6e320bb863a3a4b3a873e33d6c7613: Status 404 returned error can't find the container with id c857a961857c83087c1854f8725aadb64b6e320bb863a3a4b3a873e33d6c7613 Jul 28 22:10:43.765857 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:43.765769 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wx69k" event={"ID":"4c004d0a-52da-428b-b2f3-aefe98d15909","Type":"ContainerStarted","Data":"c857a961857c83087c1854f8725aadb64b6e320bb863a3a4b3a873e33d6c7613"} Jul 28 22:10:46.036325 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:46.036286 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/59a1f798-88a8-4236-82f2-0a675d85920d-original-pull-secret\") pod \"global-pull-secret-syncer-djzl8\" (UID: \"59a1f798-88a8-4236-82f2-0a675d85920d\") " pod="kube-system/global-pull-secret-syncer-djzl8" Jul 28 22:10:46.036790 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:46.036389 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m97d\" (UniqueName: \"kubernetes.io/projected/3b595cc6-bee1-476f-842c-00ed98dd3e79-kube-api-access-8m97d\") pod \"network-check-target-qz6gw\" (UID: \"3b595cc6-bee1-476f-842c-00ed98dd3e79\") " pod="openshift-network-diagnostics/network-check-target-qz6gw" Jul 28 22:10:46.039005 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:46.038983 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 28 22:10:46.039165 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:46.039147 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m97d\" (UniqueName: \"kubernetes.io/projected/3b595cc6-bee1-476f-842c-00ed98dd3e79-kube-api-access-8m97d\") pod \"network-check-target-qz6gw\" (UID: \"3b595cc6-bee1-476f-842c-00ed98dd3e79\") " pod="openshift-network-diagnostics/network-check-target-qz6gw" Jul 28 22:10:46.049015 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:46.048989 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/59a1f798-88a8-4236-82f2-0a675d85920d-original-pull-secret\") pod \"global-pull-secret-syncer-djzl8\" (UID: \"59a1f798-88a8-4236-82f2-0a675d85920d\") " pod="kube-system/global-pull-secret-syncer-djzl8" Jul 28 22:10:46.112293 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:46.112250 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-djzl8" Jul 28 22:10:46.120511 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:46.120488 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7dq7d\"" Jul 28 22:10:46.128976 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:46.128954 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qz6gw" Jul 28 22:10:46.137626 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:46.137600 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15e94249-e108-4426-b6f4-baf221c92a71-metrics-certs\") pod \"network-metrics-daemon-wl29s\" (UID: \"15e94249-e108-4426-b6f4-baf221c92a71\") " pod="openshift-multus/network-metrics-daemon-wl29s" Jul 28 22:10:46.139734 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:46.139715 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 28 22:10:46.151020 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:46.151001 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15e94249-e108-4426-b6f4-baf221c92a71-metrics-certs\") pod \"network-metrics-daemon-wl29s\" (UID: \"15e94249-e108-4426-b6f4-baf221c92a71\") " pod="openshift-multus/network-metrics-daemon-wl29s" Jul 28 22:10:46.328244 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:46.328156 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-q7z5n\"" Jul 28 22:10:46.336096 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:46.336066 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wl29s" Jul 28 22:10:47.248520 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:47.248479 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/139cae0a-1747-4e84-b210-24cac8e94541-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-zn5fm\" (UID: \"139cae0a-1747-4e84-b210-24cac8e94541\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-zn5fm" Jul 28 22:10:47.248520 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:47.248519 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e266916-99eb-4807-9b85-fabdeb9e1c28-metrics-certs\") pod \"router-default-6747b7576c-fs2bd\" (UID: \"7e266916-99eb-4807-9b85-fabdeb9e1c28\") " pod="openshift-ingress/router-default-6747b7576c-fs2bd" Jul 28 22:10:47.249110 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:47.248570 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4529840b-b684-4b37-b9ab-b066f7456f1b-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-ncbpn\" (UID: \"4529840b-b684-4b37-b9ab-b066f7456f1b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-ncbpn" Jul 28 22:10:47.249110 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:47.248605 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eed632c4-ac4f-4f76-8a0e-a12bcf00835d-registry-tls\") pod \"image-registry-65465d7954-dqhtb\" (UID: \"eed632c4-ac4f-4f76-8a0e-a12bcf00835d\") " pod="openshift-image-registry/image-registry-65465d7954-dqhtb" Jul 28 22:10:47.249110 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:47.248635 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a2d6df48-f09b-4f10-b909-dedec2d5b3d0-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-8nnqd\" (UID: \"a2d6df48-f09b-4f10-b909-dedec2d5b3d0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-8nnqd" Jul 28 22:10:47.249110 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:47.248666 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e266916-99eb-4807-9b85-fabdeb9e1c28-service-ca-bundle\") pod \"router-default-6747b7576c-fs2bd\" (UID: \"7e266916-99eb-4807-9b85-fabdeb9e1c28\") " pod="openshift-ingress/router-default-6747b7576c-fs2bd" Jul 28 22:10:47.249110 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:47.248870 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eb1d24ec-acde-48ed-9004-aa372bc53057-metrics-tls\") pod \"dns-default-l6gf5\" (UID: \"eb1d24ec-acde-48ed-9004-aa372bc53057\") " pod="openshift-dns/dns-default-l6gf5" Jul 28 22:10:47.249528 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:47.249362 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e266916-99eb-4807-9b85-fabdeb9e1c28-service-ca-bundle\") pod \"router-default-6747b7576c-fs2bd\" (UID: \"7e266916-99eb-4807-9b85-fabdeb9e1c28\") " pod="openshift-ingress/router-default-6747b7576c-fs2bd" Jul 28 22:10:47.251290 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:47.251235 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/139cae0a-1747-4e84-b210-24cac8e94541-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-zn5fm\" (UID: \"139cae0a-1747-4e84-b210-24cac8e94541\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-zn5fm" Jul 28 22:10:47.251439 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:47.251415 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eed632c4-ac4f-4f76-8a0e-a12bcf00835d-registry-tls\") pod \"image-registry-65465d7954-dqhtb\" (UID: \"eed632c4-ac4f-4f76-8a0e-a12bcf00835d\") " pod="openshift-image-registry/image-registry-65465d7954-dqhtb" Jul 28 22:10:47.251598 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:47.251578 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eb1d24ec-acde-48ed-9004-aa372bc53057-metrics-tls\") pod \"dns-default-l6gf5\" (UID: \"eb1d24ec-acde-48ed-9004-aa372bc53057\") " pod="openshift-dns/dns-default-l6gf5" Jul 28 22:10:47.251661 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:47.251604 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a2d6df48-f09b-4f10-b909-dedec2d5b3d0-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-8nnqd\" (UID: \"a2d6df48-f09b-4f10-b909-dedec2d5b3d0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-8nnqd" Jul 28 22:10:47.251661 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:47.251615 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4529840b-b684-4b37-b9ab-b066f7456f1b-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-ncbpn\" (UID: \"4529840b-b684-4b37-b9ab-b066f7456f1b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-ncbpn" Jul 28 22:10:47.251796 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:47.251781 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e266916-99eb-4807-9b85-fabdeb9e1c28-metrics-certs\") pod \"router-default-6747b7576c-fs2bd\" (UID: \"7e266916-99eb-4807-9b85-fabdeb9e1c28\") " pod="openshift-ingress/router-default-6747b7576c-fs2bd" Jul 28 22:10:47.399281 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:47.399247 2553 scope.go:117] "RemoveContainer" containerID="6cbbf9649882f41080c11db3f7051f60a03a344840afbabb6ea6b9bc0dab4e9d" Jul 28 22:10:47.433274 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:47.433251 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hf5kg\"" Jul 28 22:10:47.441780 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:47.441762 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65465d7954-dqhtb" Jul 28 22:10:47.445964 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:47.445947 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-4rjdf\"" Jul 28 22:10:47.454117 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:47.454100 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-ncbpn" Jul 28 22:10:47.454264 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:47.454249 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-5k4gd\"" Jul 28 22:10:47.463059 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:47.463041 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-zn5fm" Jul 28 22:10:47.483391 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:47.483373 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-8wgb6\"" Jul 28 22:10:47.492159 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:47.492143 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-8nnqd" Jul 28 22:10:47.492445 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:47.492434 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-8dvgg\"" Jul 28 22:10:47.501500 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:47.501453 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6747b7576c-fs2bd" Jul 28 22:10:47.514284 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:47.514252 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vcmq5\"" Jul 28 22:10:47.522796 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:47.522759 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-l6gf5" Jul 28 22:10:48.547945 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:48.547636 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qz6gw"] Jul 28 22:10:48.565029 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:10:48.563669 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b595cc6_bee1_476f_842c_00ed98dd3e79.slice/crio-2bcff100360687bef0d28f670cdb8f534c9f5b7fbe3eca26cde6354242d657a0 WatchSource:0}: Error finding container 2bcff100360687bef0d28f670cdb8f534c9f5b7fbe3eca26cde6354242d657a0: Status 404 returned error can't find the container with id 2bcff100360687bef0d28f670cdb8f534c9f5b7fbe3eca26cde6354242d657a0 Jul 28 22:10:48.636112 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:48.635346 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-ncbpn"] Jul 28 22:10:48.637625 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:48.637186 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wl29s"] Jul 28 22:10:48.647814 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:48.647470 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-8nnqd"] Jul 28 22:10:48.654616 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:48.653741 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-djzl8"] Jul 28 22:10:48.663458 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:10:48.663372 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4529840b_b684_4b37_b9ab_b066f7456f1b.slice/crio-e71516acad3f91c40bd5621ef5b78de3096607f4e3fa733d411d098eacfc52d5 WatchSource:0}: Error finding container e71516acad3f91c40bd5621ef5b78de3096607f4e3fa733d411d098eacfc52d5: Status 404 returned error can't find the container with id e71516acad3f91c40bd5621ef5b78de3096607f4e3fa733d411d098eacfc52d5 Jul 28 22:10:48.663650 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:10:48.663626 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59a1f798_88a8_4236_82f2_0a675d85920d.slice/crio-5dc5b884ca4d3aeba4bef6fb31c5279596560a2778128814c1af505c6ce3008d WatchSource:0}: Error finding container 5dc5b884ca4d3aeba4bef6fb31c5279596560a2778128814c1af505c6ce3008d: Status 404 returned error can't find the container with id 5dc5b884ca4d3aeba4bef6fb31c5279596560a2778128814c1af505c6ce3008d Jul 28 22:10:48.682694 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:48.682433 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-65465d7954-dqhtb"] Jul 28 22:10:48.705303 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:48.705282 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-l6gf5"] Jul 28 22:10:48.708510 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:10:48.708483 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb1d24ec_acde_48ed_9004_aa372bc53057.slice/crio-4730aeca6e03b189fd11f42047b2750de85d37186873da684bc08763cd7d7434 WatchSource:0}: Error finding container 4730aeca6e03b189fd11f42047b2750de85d37186873da684bc08763cd7d7434: Status 404 returned error can't find the container with id 4730aeca6e03b189fd11f42047b2750de85d37186873da684bc08763cd7d7434 Jul 28 22:10:48.729188 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:48.729160 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-zn5fm"] Jul 28 22:10:48.732167 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:10:48.732144 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod139cae0a_1747_4e84_b210_24cac8e94541.slice/crio-679c1e44b2ea4734a21ea6b5e4cd71be89e6f4046c4c9561e15fbaaa3263e01e WatchSource:0}: Error finding container 679c1e44b2ea4734a21ea6b5e4cd71be89e6f4046c4c9561e15fbaaa3263e01e: Status 404 returned error can't find the container with id 679c1e44b2ea4734a21ea6b5e4cd71be89e6f4046c4c9561e15fbaaa3263e01e Jul 28 22:10:48.785413 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:48.785380 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-l6gf5" event={"ID":"eb1d24ec-acde-48ed-9004-aa372bc53057","Type":"ContainerStarted","Data":"4730aeca6e03b189fd11f42047b2750de85d37186873da684bc08763cd7d7434"} Jul 28 22:10:48.786485 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:48.786459 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-8nnqd" event={"ID":"a2d6df48-f09b-4f10-b909-dedec2d5b3d0","Type":"ContainerStarted","Data":"a2c75049b00128ed8deb1b02ce671e5ca10bbd0a2d4641564070376f0bb0a0e1"} Jul 28 22:10:48.787591 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:48.787569 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-djzl8" event={"ID":"59a1f798-88a8-4236-82f2-0a675d85920d","Type":"ContainerStarted","Data":"5dc5b884ca4d3aeba4bef6fb31c5279596560a2778128814c1af505c6ce3008d"} Jul 28 22:10:48.789271 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:48.789254 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mldhp_e1ced996-f136-4d3e-9c85-a5b7809b4517/console-operator/1.log" Jul 28 22:10:48.789361 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:48.789348 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-mldhp" event={"ID":"e1ced996-f136-4d3e-9c85-a5b7809b4517","Type":"ContainerStarted","Data":"05f5aef0257a4c71779e7abda0eb2d1fe90d0e252c1117a93bf5768f03fa648a"} Jul 28 22:10:48.789759 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:48.789739 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-mldhp" Jul 28 22:10:48.790989 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:48.790965 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-ncbpn" event={"ID":"4529840b-b684-4b37-b9ab-b066f7456f1b","Type":"ContainerStarted","Data":"e71516acad3f91c40bd5621ef5b78de3096607f4e3fa733d411d098eacfc52d5"} Jul 28 22:10:48.792461 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:48.792442 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wx69k" event={"ID":"4c004d0a-52da-428b-b2f3-aefe98d15909","Type":"ContainerStarted","Data":"492da95a38ad6035050813edf56ae476f3517f4dbd92857780cab5c18fd46ceb"} Jul 28 22:10:48.793563 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:48.793545 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-zn5fm" event={"ID":"139cae0a-1747-4e84-b210-24cac8e94541","Type":"ContainerStarted","Data":"679c1e44b2ea4734a21ea6b5e4cd71be89e6f4046c4c9561e15fbaaa3263e01e"} Jul 28 22:10:48.795058 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:48.795037 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65465d7954-dqhtb" event={"ID":"eed632c4-ac4f-4f76-8a0e-a12bcf00835d","Type":"ContainerStarted","Data":"1744ef37338915fb8efaa20730d288c9bdbd76ebbb1e8788f4e1d6365b4efc1e"} Jul 28 22:10:48.795149 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:48.795069 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65465d7954-dqhtb" event={"ID":"eed632c4-ac4f-4f76-8a0e-a12bcf00835d","Type":"ContainerStarted","Data":"0c56ec0b3d119ece4926952a936522b66bdad6f58ebe2583c66821e141087044"} Jul 28 22:10:48.795149 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:48.795107 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-65465d7954-dqhtb" Jul 28 22:10:48.796325 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:48.796307 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wl29s" event={"ID":"15e94249-e108-4426-b6f4-baf221c92a71","Type":"ContainerStarted","Data":"a026d91fad3c155b49e75e36b12fdd16f943d639fbc7e09aa777d564947a7055"} Jul 28 22:10:48.797683 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:48.797662 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qz6gw" event={"ID":"3b595cc6-bee1-476f-842c-00ed98dd3e79","Type":"ContainerStarted","Data":"391b203c595d40c57e999a4952b2b91cdc6493314420638ecaec144ac7d80fbc"} Jul 28 22:10:48.797782 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:48.797715 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qz6gw" event={"ID":"3b595cc6-bee1-476f-842c-00ed98dd3e79","Type":"ContainerStarted","Data":"2bcff100360687bef0d28f670cdb8f534c9f5b7fbe3eca26cde6354242d657a0"} Jul 28 22:10:48.797832 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:48.797806 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qz6gw" Jul 28 22:10:48.806406 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:48.806361 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-mldhp" podStartSLOduration=42.069156762 podStartE2EDuration="49.806344516s" totalImagesPullingTime="7.737187754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:09:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:10:48.80599745 +0000 UTC m=+68.893255518" watchObservedRunningTime="2026-07-28 22:10:48.806344516 +0000 UTC m=+68.893602586" Jul 28 22:10:48.820811 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:48.820762 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qz6gw" podStartSLOduration=68.820743791 podStartE2EDuration="1m8.820743791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:09:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:10:48.819495962 +0000 UTC m=+68.906754031" watchObservedRunningTime="2026-07-28 22:10:48.820743791 +0000 UTC m=+68.908001862" Jul 28 22:10:48.837174 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:48.837118 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-65465d7954-dqhtb" podStartSLOduration=67.837100987 podStartE2EDuration="1m7.837100987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:09:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:10:48.836128792 +0000 UTC m=+68.923386874" watchObservedRunningTime="2026-07-28 22:10:48.837100987 +0000 UTC m=+68.924359054" Jul 28 22:10:48.902648 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:48.902613 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6747b7576c-fs2bd"] Jul 28 22:10:48.905403 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:10:48.905375 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e266916_99eb_4807_9b85_fabdeb9e1c28.slice/crio-e448979a7e077d406d328c81c97c27268db3943c92a27b08e279cd29a43d0810 WatchSource:0}: Error finding container e448979a7e077d406d328c81c97c27268db3943c92a27b08e279cd29a43d0810: Status 404 returned error can't find the container with id e448979a7e077d406d328c81c97c27268db3943c92a27b08e279cd29a43d0810 Jul 28 22:10:49.789824 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:49.789781 2553 patch_prober.go:28] interesting pod/console-operator-584dcb6b54-mldhp container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.10:8443/readyz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 28 22:10:49.790267 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:49.789857 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-584dcb6b54-mldhp" podUID="e1ced996-f136-4d3e-9c85-a5b7809b4517" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.10:8443/readyz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jul 28 22:10:49.809435 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:49.808915 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6747b7576c-fs2bd" event={"ID":"7e266916-99eb-4807-9b85-fabdeb9e1c28","Type":"ContainerStarted","Data":"28f33582e27ec53a288a12edfacebd2f7e8e911fe1332cebf2c8d28987966a2b"} Jul 28 22:10:49.809435 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:49.808981 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6747b7576c-fs2bd" event={"ID":"7e266916-99eb-4807-9b85-fabdeb9e1c28","Type":"ContainerStarted","Data":"e448979a7e077d406d328c81c97c27268db3943c92a27b08e279cd29a43d0810"} Jul 28 22:10:49.829387 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:49.827936 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-6747b7576c-fs2bd" podStartSLOduration=50.827900331 podStartE2EDuration="50.827900331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:09:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:10:49.827252597 +0000 UTC m=+69.914510696" watchObservedRunningTime="2026-07-28 22:10:49.827900331 +0000 UTC m=+69.915158403" Jul 28 22:10:50.504162 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:50.502907 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6747b7576c-fs2bd" Jul 28 22:10:50.506176 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:50.505981 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6747b7576c-fs2bd" Jul 28 22:10:50.809601 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:50.809513 2553 patch_prober.go:28] interesting pod/console-operator-584dcb6b54-mldhp container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.10:8443/readyz\": context deadline exceeded" start-of-body= Jul 28 22:10:50.809601 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:50.809579 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-584dcb6b54-mldhp" podUID="e1ced996-f136-4d3e-9c85-a5b7809b4517" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.10:8443/readyz\": context deadline exceeded" Jul 28 22:10:50.815379 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:50.815314 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6747b7576c-fs2bd" Jul 28 22:10:50.817092 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:50.816900 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6747b7576c-fs2bd" Jul 28 22:10:51.512880 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:51.512842 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-hcqll"] Jul 28 22:10:51.516274 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:51.516248 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-hcqll" Jul 28 22:10:51.518925 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:51.518592 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-hcqll"] Jul 28 22:10:51.518925 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:51.518684 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-d6w9c\"" Jul 28 22:10:51.518925 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:51.518776 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 28 22:10:51.518925 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:51.518915 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 28 22:10:51.697330 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:51.697291 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-586jx\" (UniqueName: \"kubernetes.io/projected/cd00fd19-681f-4de7-85ad-aea576edbac6-kube-api-access-586jx\") pod \"downloads-7c97dc4555-hcqll\" (UID: \"cd00fd19-681f-4de7-85ad-aea576edbac6\") " pod="openshift-console/downloads-7c97dc4555-hcqll" Jul 28 22:10:51.798638 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:51.798544 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-586jx\" (UniqueName: \"kubernetes.io/projected/cd00fd19-681f-4de7-85ad-aea576edbac6-kube-api-access-586jx\") pod \"downloads-7c97dc4555-hcqll\" (UID: \"cd00fd19-681f-4de7-85ad-aea576edbac6\") " pod="openshift-console/downloads-7c97dc4555-hcqll" Jul 28 22:10:51.809686 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:51.809653 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-586jx\" (UniqueName: \"kubernetes.io/projected/cd00fd19-681f-4de7-85ad-aea576edbac6-kube-api-access-586jx\") pod \"downloads-7c97dc4555-hcqll\" (UID: \"cd00fd19-681f-4de7-85ad-aea576edbac6\") " pod="openshift-console/downloads-7c97dc4555-hcqll" Jul 28 22:10:51.830177 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:51.830151 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-hcqll" Jul 28 22:10:57.043177 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:57.042565 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-hcqll"] Jul 28 22:10:57.053146 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:10:57.053038 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd00fd19_681f_4de7_85ad_aea576edbac6.slice/crio-b5cce642cab4ceccb5bf88a1569992dd3b817b6a8b2fb457b76073ecd85c61e1 WatchSource:0}: Error finding container b5cce642cab4ceccb5bf88a1569992dd3b817b6a8b2fb457b76073ecd85c61e1: Status 404 returned error can't find the container with id b5cce642cab4ceccb5bf88a1569992dd3b817b6a8b2fb457b76073ecd85c61e1 Jul 28 22:10:57.848365 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:57.848292 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-ncbpn" event={"ID":"4529840b-b684-4b37-b9ab-b066f7456f1b","Type":"ContainerStarted","Data":"defcf1aa0ff8f4b878b4fc157e12e0ceff2b63caa08753b85fed6125471156bf"} Jul 28 22:10:57.850227 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:57.850193 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-zn5fm" event={"ID":"139cae0a-1747-4e84-b210-24cac8e94541","Type":"ContainerStarted","Data":"f0a7d8da9c71083460b05ceb1586cf2e8a88cc0a8d07177e222cad6fc26e79f2"} Jul 28 22:10:57.853914 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:57.853558 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-l6gf5" event={"ID":"eb1d24ec-acde-48ed-9004-aa372bc53057","Type":"ContainerStarted","Data":"3b311f87416463ac0994e88dfce2dfef4899dadd5de94a7e1ae6d8c1bec34c88"} Jul 28 22:10:57.853914 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:57.853606 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-l6gf5" event={"ID":"eb1d24ec-acde-48ed-9004-aa372bc53057","Type":"ContainerStarted","Data":"f7e5b5f153694a9d79ee8fec10c6407b841cf293aee2955ddf81e110eadb61ac"} Jul 28 22:10:57.853914 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:57.853648 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-l6gf5" Jul 28 22:10:57.855574 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:57.855543 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-djzl8" event={"ID":"59a1f798-88a8-4236-82f2-0a675d85920d","Type":"ContainerStarted","Data":"86ee901062956e7222b989472e4b32e9fabb4c227a31f26373e3a486cd248dc1"} Jul 28 22:10:57.858047 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:57.858010 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wl29s" event={"ID":"15e94249-e108-4426-b6f4-baf221c92a71","Type":"ContainerStarted","Data":"4c31c28e9fa5f3667488254990102436d04a44d3012f2df4fe1b2870b1f3a153"} Jul 28 22:10:57.858206 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:57.858059 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wl29s" event={"ID":"15e94249-e108-4426-b6f4-baf221c92a71","Type":"ContainerStarted","Data":"c125b8dbdf33a9bf1072c30b74984c2d320c06e8fe95961fbd08ee3505d4ab16"} Jul 28 22:10:57.860381 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:57.860357 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wx69k" event={"ID":"4c004d0a-52da-428b-b2f3-aefe98d15909","Type":"ContainerStarted","Data":"40e0f4ed86672258391e95d3a80d035b79951833c0dd3f1de14e0cec7dbbca23"} Jul 28 22:10:57.863280 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:57.863251 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-8nnqd" event={"ID":"a2d6df48-f09b-4f10-b909-dedec2d5b3d0","Type":"ContainerStarted","Data":"f9692d3bc7a7d1aaf1c114f54b268fb8c4c03933e8b35c23d5bd60660de24b8c"} Jul 28 22:10:57.863753 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:57.863690 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-8nnqd" event={"ID":"a2d6df48-f09b-4f10-b909-dedec2d5b3d0","Type":"ContainerStarted","Data":"6692e6772cc271ebfe71fcbd8479e2b0118dff5cbdaa08f6b7c13c02da57aaed"} Jul 28 22:10:57.864620 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:57.864585 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-hcqll" event={"ID":"cd00fd19-681f-4de7-85ad-aea576edbac6","Type":"ContainerStarted","Data":"b5cce642cab4ceccb5bf88a1569992dd3b817b6a8b2fb457b76073ecd85c61e1"} Jul 28 22:10:57.868750 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:57.867442 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-ncbpn" podStartSLOduration=50.971323171 podStartE2EDuration="58.867425004s" totalImagesPullingTime="7.896101833s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:09:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:10:57.866781042 +0000 UTC m=+77.954039111" watchObservedRunningTime="2026-07-28 22:10:57.867425004 +0000 UTC m=+77.954683055" Jul 28 22:10:57.882489 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:57.882409 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-zn5fm" podStartSLOduration=50.719217901 podStartE2EDuration="58.882388122s" totalImagesPullingTime="8.163170221s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:09:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:10:57.881604972 +0000 UTC m=+77.968863043" watchObservedRunningTime="2026-07-28 22:10:57.882388122 +0000 UTC m=+77.969646189" Jul 28 22:10:57.898373 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:57.898315 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-8nnqd" podStartSLOduration=50.712385312 podStartE2EDuration="58.898295485s" totalImagesPullingTime="8.185910173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:09:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:10:57.896200259 +0000 UTC m=+77.983458329" watchObservedRunningTime="2026-07-28 22:10:57.898295485 +0000 UTC m=+77.985553558" Jul 28 22:10:57.917271 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:57.916379 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-l6gf5" podStartSLOduration=34.728409462 podStartE2EDuration="42.916358184s" totalImagesPullingTime="8.187948722s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:10:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:10:57.914755445 +0000 UTC m=+78.002013514" watchObservedRunningTime="2026-07-28 22:10:57.916358184 +0000 UTC m=+78.003616255" Jul 28 22:10:57.930177 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:57.930087 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-djzl8" podStartSLOduration=69.697358291 podStartE2EDuration="1m17.930063239s" totalImagesPullingTime="8.232704948s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 22:09:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:10:57.928142116 +0000 UTC m=+78.015400185" watchObservedRunningTime="2026-07-28 22:10:57.930063239 +0000 UTC m=+78.017321311" Jul 28 22:10:57.944020 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:57.942674 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-wl29s" podStartSLOduration=70.733596206 podStartE2EDuration="1m17.942651081s" totalImagesPullingTime="7.209054875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:09:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:10:57.942077958 +0000 UTC m=+78.029336038" watchObservedRunningTime="2026-07-28 22:10:57.942651081 +0000 UTC m=+78.029909150" Jul 28 22:10:59.814623 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:10:59.814587 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-mldhp" Jul 28 22:11:01.886618 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:01.886580 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wx69k" event={"ID":"4c004d0a-52da-428b-b2f3-aefe98d15909","Type":"ContainerStarted","Data":"4c6a2fdf3a05d88b4b08a4cbaafbe5538972ce62a0cfb65fce6cc4fd7f133d5f"} Jul 28 22:11:01.907520 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:01.907436 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-wx69k" podStartSLOduration=17.33687988 podStartE2EDuration="35.907414856s" totalImagesPullingTime="18.570534976s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 22:10:26 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:11:01.90710618 +0000 UTC m=+81.994364265" watchObservedRunningTime="2026-07-28 22:11:01.907414856 +0000 UTC m=+81.994672925" Jul 28 22:11:07.447262 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:07.447205 2553 patch_prober.go:28] interesting pod/image-registry-65465d7954-dqhtb container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 28 22:11:07.447823 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:07.447274 2553 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-65465d7954-dqhtb" podUID="eed632c4-ac4f-4f76-8a0e-a12bcf00835d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 22:11:07.874281 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:07.874240 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-l6gf5" Jul 28 22:11:09.815127 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:09.815090 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-65465d7954-dqhtb" Jul 28 22:11:11.022415 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:11.022382 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vnzs2"] Jul 28 22:11:11.061320 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:11.061289 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vnzs2" Jul 28 22:11:11.064394 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:11.064306 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-n8cxc\"" Jul 28 22:11:11.064394 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:11.064335 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 28 22:11:11.064394 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:11.064375 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 28 22:11:11.064581 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:11.064431 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 28 22:11:11.064746 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:11.064729 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 28 22:11:11.153655 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:11.153620 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b7760431-5b30-439d-83dc-f5fdaec65569-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vnzs2\" (UID: \"b7760431-5b30-439d-83dc-f5fdaec65569\") " pod="openshift-monitoring/node-exporter-vnzs2" Jul 28 22:11:11.153841 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:11.153671 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b7760431-5b30-439d-83dc-f5fdaec65569-node-exporter-tls\") pod \"node-exporter-vnzs2\" (UID: \"b7760431-5b30-439d-83dc-f5fdaec65569\") " pod="openshift-monitoring/node-exporter-vnzs2" Jul 28 22:11:11.153841 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:11.153718 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b7760431-5b30-439d-83dc-f5fdaec65569-node-exporter-wtmp\") pod \"node-exporter-vnzs2\" (UID: \"b7760431-5b30-439d-83dc-f5fdaec65569\") " pod="openshift-monitoring/node-exporter-vnzs2" Jul 28 22:11:11.153841 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:11.153777 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b7760431-5b30-439d-83dc-f5fdaec65569-node-exporter-accelerators-collector-config\") pod \"node-exporter-vnzs2\" (UID: \"b7760431-5b30-439d-83dc-f5fdaec65569\") " pod="openshift-monitoring/node-exporter-vnzs2" Jul 28 22:11:11.154000 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:11.153961 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xqn5\" (UniqueName: \"kubernetes.io/projected/b7760431-5b30-439d-83dc-f5fdaec65569-kube-api-access-8xqn5\") pod \"node-exporter-vnzs2\" (UID: \"b7760431-5b30-439d-83dc-f5fdaec65569\") " pod="openshift-monitoring/node-exporter-vnzs2" Jul 28 22:11:11.154062 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:11.154039 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7760431-5b30-439d-83dc-f5fdaec65569-sys\") pod \"node-exporter-vnzs2\" (UID: \"b7760431-5b30-439d-83dc-f5fdaec65569\") " pod="openshift-monitoring/node-exporter-vnzs2" Jul 28 22:11:11.154122 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:11.154071 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b7760431-5b30-439d-83dc-f5fdaec65569-metrics-client-ca\") pod \"node-exporter-vnzs2\" (UID: \"b7760431-5b30-439d-83dc-f5fdaec65569\") " pod="openshift-monitoring/node-exporter-vnzs2" Jul 28 22:11:11.154122 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:11.154099 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b7760431-5b30-439d-83dc-f5fdaec65569-node-exporter-textfile\") pod \"node-exporter-vnzs2\" (UID: \"b7760431-5b30-439d-83dc-f5fdaec65569\") " pod="openshift-monitoring/node-exporter-vnzs2" Jul 28 22:11:11.154213 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:11.154128 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b7760431-5b30-439d-83dc-f5fdaec65569-root\") pod \"node-exporter-vnzs2\" (UID: \"b7760431-5b30-439d-83dc-f5fdaec65569\") " pod="openshift-monitoring/node-exporter-vnzs2" Jul 28 22:11:11.254758 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:11.254691 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7760431-5b30-439d-83dc-f5fdaec65569-sys\") pod \"node-exporter-vnzs2\" (UID: \"b7760431-5b30-439d-83dc-f5fdaec65569\") " pod="openshift-monitoring/node-exporter-vnzs2" Jul 28 22:11:11.254950 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:11.254775 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b7760431-5b30-439d-83dc-f5fdaec65569-metrics-client-ca\") pod \"node-exporter-vnzs2\" (UID: \"b7760431-5b30-439d-83dc-f5fdaec65569\") " pod="openshift-monitoring/node-exporter-vnzs2" Jul 28 22:11:11.254950 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:11.254804 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b7760431-5b30-439d-83dc-f5fdaec65569-node-exporter-textfile\") pod \"node-exporter-vnzs2\" (UID: \"b7760431-5b30-439d-83dc-f5fdaec65569\") " pod="openshift-monitoring/node-exporter-vnzs2" Jul 28 22:11:11.254950 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:11.254827 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7760431-5b30-439d-83dc-f5fdaec65569-sys\") pod \"node-exporter-vnzs2\" (UID: \"b7760431-5b30-439d-83dc-f5fdaec65569\") " pod="openshift-monitoring/node-exporter-vnzs2" Jul 28 22:11:11.254950 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:11.254892 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b7760431-5b30-439d-83dc-f5fdaec65569-root\") pod \"node-exporter-vnzs2\" (UID: \"b7760431-5b30-439d-83dc-f5fdaec65569\") " pod="openshift-monitoring/node-exporter-vnzs2" Jul 28 22:11:11.254950 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:11.254834 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b7760431-5b30-439d-83dc-f5fdaec65569-root\") pod \"node-exporter-vnzs2\" (UID: \"b7760431-5b30-439d-83dc-f5fdaec65569\") " pod="openshift-monitoring/node-exporter-vnzs2" Jul 28 22:11:11.254950 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:11.254943 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b7760431-5b30-439d-83dc-f5fdaec65569-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vnzs2\" (UID: \"b7760431-5b30-439d-83dc-f5fdaec65569\") " pod="openshift-monitoring/node-exporter-vnzs2" Jul 28 22:11:11.255250 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:11.254976 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b7760431-5b30-439d-83dc-f5fdaec65569-node-exporter-tls\") pod \"node-exporter-vnzs2\" (UID: \"b7760431-5b30-439d-83dc-f5fdaec65569\") " pod="openshift-monitoring/node-exporter-vnzs2" Jul 28 22:11:11.255250 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:11.254999 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b7760431-5b30-439d-83dc-f5fdaec65569-node-exporter-wtmp\") pod \"node-exporter-vnzs2\" (UID: \"b7760431-5b30-439d-83dc-f5fdaec65569\") " pod="openshift-monitoring/node-exporter-vnzs2" Jul 28 22:11:11.255250 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:11.255024 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b7760431-5b30-439d-83dc-f5fdaec65569-node-exporter-accelerators-collector-config\") pod \"node-exporter-vnzs2\" (UID: \"b7760431-5b30-439d-83dc-f5fdaec65569\") " pod="openshift-monitoring/node-exporter-vnzs2" Jul 28 22:11:11.255250 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:11.255124 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xqn5\" (UniqueName: \"kubernetes.io/projected/b7760431-5b30-439d-83dc-f5fdaec65569-kube-api-access-8xqn5\") pod \"node-exporter-vnzs2\" (UID: \"b7760431-5b30-439d-83dc-f5fdaec65569\") " pod="openshift-monitoring/node-exporter-vnzs2" Jul 28 22:11:11.255250 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:11.255216 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b7760431-5b30-439d-83dc-f5fdaec65569-node-exporter-textfile\") pod \"node-exporter-vnzs2\" (UID: \"b7760431-5b30-439d-83dc-f5fdaec65569\") " pod="openshift-monitoring/node-exporter-vnzs2" Jul 28 22:11:11.255508 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:11.255248 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b7760431-5b30-439d-83dc-f5fdaec65569-node-exporter-wtmp\") pod \"node-exporter-vnzs2\" (UID: \"b7760431-5b30-439d-83dc-f5fdaec65569\") " pod="openshift-monitoring/node-exporter-vnzs2" Jul 28 22:11:11.255508 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:11.255484 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b7760431-5b30-439d-83dc-f5fdaec65569-metrics-client-ca\") pod \"node-exporter-vnzs2\" (UID: \"b7760431-5b30-439d-83dc-f5fdaec65569\") " pod="openshift-monitoring/node-exporter-vnzs2" Jul 28 22:11:11.255725 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:11.255678 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b7760431-5b30-439d-83dc-f5fdaec65569-node-exporter-accelerators-collector-config\") pod \"node-exporter-vnzs2\" (UID: \"b7760431-5b30-439d-83dc-f5fdaec65569\") " pod="openshift-monitoring/node-exporter-vnzs2" Jul 28 22:11:11.257884 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:11.257862 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b7760431-5b30-439d-83dc-f5fdaec65569-node-exporter-tls\") pod \"node-exporter-vnzs2\" (UID: \"b7760431-5b30-439d-83dc-f5fdaec65569\") " pod="openshift-monitoring/node-exporter-vnzs2" Jul 28 22:11:11.258180 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:11.258161 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b7760431-5b30-439d-83dc-f5fdaec65569-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vnzs2\" (UID: \"b7760431-5b30-439d-83dc-f5fdaec65569\") " pod="openshift-monitoring/node-exporter-vnzs2" Jul 28 22:11:11.263640 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:11.263615 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xqn5\" (UniqueName: \"kubernetes.io/projected/b7760431-5b30-439d-83dc-f5fdaec65569-kube-api-access-8xqn5\") pod \"node-exporter-vnzs2\" (UID: \"b7760431-5b30-439d-83dc-f5fdaec65569\") " pod="openshift-monitoring/node-exporter-vnzs2" Jul 28 22:11:11.372026 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:11.371938 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vnzs2" Jul 28 22:11:14.460020 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:11:14.459987 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7760431_5b30_439d_83dc_f5fdaec65569.slice/crio-df1bf48bfc88db82d83d5e6ddef8fee086fb5d4db5d9f02fd4dd5804cdaa8b71 WatchSource:0}: Error finding container df1bf48bfc88db82d83d5e6ddef8fee086fb5d4db5d9f02fd4dd5804cdaa8b71: Status 404 returned error can't find the container with id df1bf48bfc88db82d83d5e6ddef8fee086fb5d4db5d9f02fd4dd5804cdaa8b71 Jul 28 22:11:14.950406 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:14.950365 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-hcqll" event={"ID":"cd00fd19-681f-4de7-85ad-aea576edbac6","Type":"ContainerStarted","Data":"410f54e74dd5a03e960e4c462a6b7b01dbdd243f09b6fdd379ad156ac2d35f13"} Jul 28 22:11:14.950746 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:14.950720 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-hcqll" Jul 28 22:11:14.951927 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:14.951902 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vnzs2" event={"ID":"b7760431-5b30-439d-83dc-f5fdaec65569","Type":"ContainerStarted","Data":"df1bf48bfc88db82d83d5e6ddef8fee086fb5d4db5d9f02fd4dd5804cdaa8b71"} Jul 28 22:11:14.952242 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:14.952222 2553 patch_prober.go:28] interesting pod/downloads-7c97dc4555-hcqll container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.132.0.21:8080/\": dial tcp 10.132.0.21:8080: connect: connection refused" start-of-body= Jul 28 22:11:14.952339 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:14.952282 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-7c97dc4555-hcqll" podUID="cd00fd19-681f-4de7-85ad-aea576edbac6" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.21:8080/\": dial tcp 10.132.0.21:8080: connect: connection refused" Jul 28 22:11:14.967441 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:14.967393 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-hcqll" podStartSLOduration=6.220848197 podStartE2EDuration="23.967379775s" totalImagesPullingTime="17.746531578s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:10:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:11:14.966128737 +0000 UTC m=+95.053386811" watchObservedRunningTime="2026-07-28 22:11:14.967379775 +0000 UTC m=+95.054637843" Jul 28 22:11:15.958960 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:15.958920 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vnzs2" event={"ID":"b7760431-5b30-439d-83dc-f5fdaec65569","Type":"ContainerStarted","Data":"ce8f87a79438ae0a49c9e1b66ef8ba529eee7d5d6f40bf6c5f185b6a07a966e6"} Jul 28 22:11:15.966430 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:15.965967 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-65465d7954-dqhtb"] Jul 28 22:11:15.969730 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:15.969633 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-hcqll" Jul 28 22:11:16.965181 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:16.965148 2553 generic.go:356] "Generic (PLEG): container finished" podID="b7760431-5b30-439d-83dc-f5fdaec65569" containerID="ce8f87a79438ae0a49c9e1b66ef8ba529eee7d5d6f40bf6c5f185b6a07a966e6" exitCode=0 Jul 28 22:11:16.965690 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:16.965256 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vnzs2" event={"ID":"b7760431-5b30-439d-83dc-f5fdaec65569","Type":"ContainerDied","Data":"ce8f87a79438ae0a49c9e1b66ef8ba529eee7d5d6f40bf6c5f185b6a07a966e6"} Jul 28 22:11:17.972123 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:17.972079 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vnzs2" event={"ID":"b7760431-5b30-439d-83dc-f5fdaec65569","Type":"ContainerStarted","Data":"f2df85d16b5b50dc35769266d8e32a8707b9e20540bc34b91a3d03d1a307d975"} Jul 28 22:11:17.972563 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:17.972151 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vnzs2" event={"ID":"b7760431-5b30-439d-83dc-f5fdaec65569","Type":"ContainerStarted","Data":"ff83416228b59a31ad622e345adb3c4842648b97d695e909a230b96c1a339230"} Jul 28 22:11:17.993258 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:17.993203 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vnzs2" podStartSLOduration=5.57025501 podStartE2EDuration="6.993182682s" totalImagesPullingTime="1.305122182s" totalInitContainerRuntime="117.80549ms" isStatefulPod=true podCreationTimestamp="2026-07-28 22:11:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:11:17.992636723 +0000 UTC m=+98.079894792" watchObservedRunningTime="2026-07-28 22:11:17.993182682 +0000 UTC m=+98.080440751" Jul 28 22:11:19.812515 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:19.812432 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qz6gw" Jul 28 22:11:20.812200 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:20.812163 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-d86f68db7-v2w7n"] Jul 28 22:11:20.879516 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:20.879476 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d86f68db7-v2w7n"] Jul 28 22:11:20.880022 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:20.879608 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d86f68db7-v2w7n" Jul 28 22:11:20.882382 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:20.882361 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 28 22:11:20.882591 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:20.882566 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-jr9lx\"" Jul 28 22:11:20.882591 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:20.882366 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 28 22:11:20.882799 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:20.882500 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 28 22:11:20.883046 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:20.883020 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 28 22:11:20.883046 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:20.883024 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 28 22:11:20.887624 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:20.887589 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 28 22:11:20.950072 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:20.950036 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f1169058-aba8-4b2b-9d11-23cb12a61ff1-console-serving-cert\") pod \"console-d86f68db7-v2w7n\" (UID: \"f1169058-aba8-4b2b-9d11-23cb12a61ff1\") " pod="openshift-console/console-d86f68db7-v2w7n" Jul 28 22:11:20.950314 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:20.950289 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f1169058-aba8-4b2b-9d11-23cb12a61ff1-console-oauth-config\") pod \"console-d86f68db7-v2w7n\" (UID: \"f1169058-aba8-4b2b-9d11-23cb12a61ff1\") " pod="openshift-console/console-d86f68db7-v2w7n" Jul 28 22:11:20.950408 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:20.950390 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f1169058-aba8-4b2b-9d11-23cb12a61ff1-console-config\") pod \"console-d86f68db7-v2w7n\" (UID: \"f1169058-aba8-4b2b-9d11-23cb12a61ff1\") " pod="openshift-console/console-d86f68db7-v2w7n" Jul 28 22:11:20.950461 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:20.950425 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1169058-aba8-4b2b-9d11-23cb12a61ff1-trusted-ca-bundle\") pod \"console-d86f68db7-v2w7n\" (UID: \"f1169058-aba8-4b2b-9d11-23cb12a61ff1\") " pod="openshift-console/console-d86f68db7-v2w7n" Jul 28 22:11:20.950546 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:20.950523 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f1169058-aba8-4b2b-9d11-23cb12a61ff1-service-ca\") pod \"console-d86f68db7-v2w7n\" (UID: \"f1169058-aba8-4b2b-9d11-23cb12a61ff1\") " pod="openshift-console/console-d86f68db7-v2w7n" Jul 28 22:11:20.950604 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:20.950563 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f1169058-aba8-4b2b-9d11-23cb12a61ff1-oauth-serving-cert\") pod \"console-d86f68db7-v2w7n\" (UID: \"f1169058-aba8-4b2b-9d11-23cb12a61ff1\") " pod="openshift-console/console-d86f68db7-v2w7n" Jul 28 22:11:20.950604 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:20.950592 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj26c\" (UniqueName: \"kubernetes.io/projected/f1169058-aba8-4b2b-9d11-23cb12a61ff1-kube-api-access-cj26c\") pod \"console-d86f68db7-v2w7n\" (UID: \"f1169058-aba8-4b2b-9d11-23cb12a61ff1\") " pod="openshift-console/console-d86f68db7-v2w7n" Jul 28 22:11:21.051679 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:21.051637 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f1169058-aba8-4b2b-9d11-23cb12a61ff1-console-config\") pod \"console-d86f68db7-v2w7n\" (UID: \"f1169058-aba8-4b2b-9d11-23cb12a61ff1\") " pod="openshift-console/console-d86f68db7-v2w7n" Jul 28 22:11:21.051679 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:21.051683 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1169058-aba8-4b2b-9d11-23cb12a61ff1-trusted-ca-bundle\") pod \"console-d86f68db7-v2w7n\" (UID: \"f1169058-aba8-4b2b-9d11-23cb12a61ff1\") " pod="openshift-console/console-d86f68db7-v2w7n" Jul 28 22:11:21.051942 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:21.051757 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f1169058-aba8-4b2b-9d11-23cb12a61ff1-service-ca\") pod \"console-d86f68db7-v2w7n\" (UID: \"f1169058-aba8-4b2b-9d11-23cb12a61ff1\") " pod="openshift-console/console-d86f68db7-v2w7n" Jul 28 22:11:21.051942 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:21.051774 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f1169058-aba8-4b2b-9d11-23cb12a61ff1-oauth-serving-cert\") pod \"console-d86f68db7-v2w7n\" (UID: \"f1169058-aba8-4b2b-9d11-23cb12a61ff1\") " pod="openshift-console/console-d86f68db7-v2w7n" Jul 28 22:11:21.051942 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:21.051796 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj26c\" (UniqueName: \"kubernetes.io/projected/f1169058-aba8-4b2b-9d11-23cb12a61ff1-kube-api-access-cj26c\") pod \"console-d86f68db7-v2w7n\" (UID: \"f1169058-aba8-4b2b-9d11-23cb12a61ff1\") " pod="openshift-console/console-d86f68db7-v2w7n" Jul 28 22:11:21.051942 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:21.051829 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f1169058-aba8-4b2b-9d11-23cb12a61ff1-console-serving-cert\") pod \"console-d86f68db7-v2w7n\" (UID: \"f1169058-aba8-4b2b-9d11-23cb12a61ff1\") " pod="openshift-console/console-d86f68db7-v2w7n" Jul 28 22:11:21.051942 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:21.051851 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f1169058-aba8-4b2b-9d11-23cb12a61ff1-console-oauth-config\") pod \"console-d86f68db7-v2w7n\" (UID: \"f1169058-aba8-4b2b-9d11-23cb12a61ff1\") " pod="openshift-console/console-d86f68db7-v2w7n" Jul 28 22:11:21.052504 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:21.052474 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f1169058-aba8-4b2b-9d11-23cb12a61ff1-console-config\") pod \"console-d86f68db7-v2w7n\" (UID: \"f1169058-aba8-4b2b-9d11-23cb12a61ff1\") " pod="openshift-console/console-d86f68db7-v2w7n" Jul 28 22:11:21.052603 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:21.052483 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f1169058-aba8-4b2b-9d11-23cb12a61ff1-service-ca\") pod \"console-d86f68db7-v2w7n\" (UID: \"f1169058-aba8-4b2b-9d11-23cb12a61ff1\") " pod="openshift-console/console-d86f68db7-v2w7n" Jul 28 22:11:21.053119 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:21.053088 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f1169058-aba8-4b2b-9d11-23cb12a61ff1-oauth-serving-cert\") pod \"console-d86f68db7-v2w7n\" (UID: \"f1169058-aba8-4b2b-9d11-23cb12a61ff1\") " pod="openshift-console/console-d86f68db7-v2w7n" Jul 28 22:11:21.053235 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:21.053213 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1169058-aba8-4b2b-9d11-23cb12a61ff1-trusted-ca-bundle\") pod \"console-d86f68db7-v2w7n\" (UID: \"f1169058-aba8-4b2b-9d11-23cb12a61ff1\") " pod="openshift-console/console-d86f68db7-v2w7n" Jul 28 22:11:21.054798 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:21.054779 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f1169058-aba8-4b2b-9d11-23cb12a61ff1-console-oauth-config\") pod \"console-d86f68db7-v2w7n\" (UID: \"f1169058-aba8-4b2b-9d11-23cb12a61ff1\") " pod="openshift-console/console-d86f68db7-v2w7n" Jul 28 22:11:21.055207 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:21.055185 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f1169058-aba8-4b2b-9d11-23cb12a61ff1-console-serving-cert\") pod \"console-d86f68db7-v2w7n\" (UID: \"f1169058-aba8-4b2b-9d11-23cb12a61ff1\") " pod="openshift-console/console-d86f68db7-v2w7n" Jul 28 22:11:21.062278 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:21.062216 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj26c\" (UniqueName: \"kubernetes.io/projected/f1169058-aba8-4b2b-9d11-23cb12a61ff1-kube-api-access-cj26c\") pod \"console-d86f68db7-v2w7n\" (UID: \"f1169058-aba8-4b2b-9d11-23cb12a61ff1\") " pod="openshift-console/console-d86f68db7-v2w7n" Jul 28 22:11:21.191369 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:21.191330 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d86f68db7-v2w7n" Jul 28 22:11:21.381814 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:21.381654 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d86f68db7-v2w7n"] Jul 28 22:11:21.384845 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:11:21.384792 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1169058_aba8_4b2b_9d11_23cb12a61ff1.slice/crio-fa493fc6b75bdb011b90e264c1ce1eefdf460ffb638360258cc2f012e5c441f9 WatchSource:0}: Error finding container fa493fc6b75bdb011b90e264c1ce1eefdf460ffb638360258cc2f012e5c441f9: Status 404 returned error can't find the container with id fa493fc6b75bdb011b90e264c1ce1eefdf460ffb638360258cc2f012e5c441f9 Jul 28 22:11:21.998233 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:21.998178 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d86f68db7-v2w7n" event={"ID":"f1169058-aba8-4b2b-9d11-23cb12a61ff1","Type":"ContainerStarted","Data":"fa493fc6b75bdb011b90e264c1ce1eefdf460ffb638360258cc2f012e5c441f9"} Jul 28 22:11:26.017323 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:26.017280 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d86f68db7-v2w7n" event={"ID":"f1169058-aba8-4b2b-9d11-23cb12a61ff1","Type":"ContainerStarted","Data":"6528fd0fd515fe87256262c3ede663d5882afc97510fa86c849ce5b95f8b097a"} Jul 28 22:11:26.036139 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:26.036088 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-d86f68db7-v2w7n" podStartSLOduration=2.292593004 podStartE2EDuration="6.0360727s" totalImagesPullingTime="3.743479696s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:11:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:11:26.034585895 +0000 UTC m=+106.121843984" watchObservedRunningTime="2026-07-28 22:11:26.0360727 +0000 UTC m=+106.123330769" Jul 28 22:11:31.192469 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:31.192426 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-d86f68db7-v2w7n" Jul 28 22:11:31.193020 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:31.192537 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-d86f68db7-v2w7n" Jul 28 22:11:31.197258 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:31.197239 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-d86f68db7-v2w7n" Jul 28 22:11:32.044085 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:32.044054 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-d86f68db7-v2w7n" Jul 28 22:11:40.991756 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:40.991690 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-65465d7954-dqhtb" podUID="eed632c4-ac4f-4f76-8a0e-a12bcf00835d" containerName="registry" containerID="cri-o://1744ef37338915fb8efaa20730d288c9bdbd76ebbb1e8788f4e1d6365b4efc1e" gracePeriod=30 Jul 28 22:11:41.276803 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:41.276740 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65465d7954-dqhtb" Jul 28 22:11:41.435872 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:41.435843 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/eed632c4-ac4f-4f76-8a0e-a12bcf00835d-installation-pull-secrets\") pod \"eed632c4-ac4f-4f76-8a0e-a12bcf00835d\" (UID: \"eed632c4-ac4f-4f76-8a0e-a12bcf00835d\") " Jul 28 22:11:41.436024 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:41.435892 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/eed632c4-ac4f-4f76-8a0e-a12bcf00835d-image-registry-private-configuration\") pod \"eed632c4-ac4f-4f76-8a0e-a12bcf00835d\" (UID: \"eed632c4-ac4f-4f76-8a0e-a12bcf00835d\") " Jul 28 22:11:41.436024 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:41.435922 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eed632c4-ac4f-4f76-8a0e-a12bcf00835d-registry-tls\") pod \"eed632c4-ac4f-4f76-8a0e-a12bcf00835d\" (UID: \"eed632c4-ac4f-4f76-8a0e-a12bcf00835d\") " Jul 28 22:11:41.436024 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:41.435950 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eed632c4-ac4f-4f76-8a0e-a12bcf00835d-trusted-ca\") pod \"eed632c4-ac4f-4f76-8a0e-a12bcf00835d\" (UID: \"eed632c4-ac4f-4f76-8a0e-a12bcf00835d\") " Jul 28 22:11:41.436024 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:41.436003 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/eed632c4-ac4f-4f76-8a0e-a12bcf00835d-registry-certificates\") pod \"eed632c4-ac4f-4f76-8a0e-a12bcf00835d\" (UID: \"eed632c4-ac4f-4f76-8a0e-a12bcf00835d\") " Jul 28 22:11:41.436241 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:41.436138 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/eed632c4-ac4f-4f76-8a0e-a12bcf00835d-ca-trust-extracted\") pod \"eed632c4-ac4f-4f76-8a0e-a12bcf00835d\" (UID: \"eed632c4-ac4f-4f76-8a0e-a12bcf00835d\") " Jul 28 22:11:41.436241 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:41.436191 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eed632c4-ac4f-4f76-8a0e-a12bcf00835d-bound-sa-token\") pod \"eed632c4-ac4f-4f76-8a0e-a12bcf00835d\" (UID: \"eed632c4-ac4f-4f76-8a0e-a12bcf00835d\") " Jul 28 22:11:41.436241 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:41.436220 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xgb67\" (UniqueName: \"kubernetes.io/projected/eed632c4-ac4f-4f76-8a0e-a12bcf00835d-kube-api-access-xgb67\") pod \"eed632c4-ac4f-4f76-8a0e-a12bcf00835d\" (UID: \"eed632c4-ac4f-4f76-8a0e-a12bcf00835d\") " Jul 28 22:11:41.436534 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:41.436490 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eed632c4-ac4f-4f76-8a0e-a12bcf00835d-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "eed632c4-ac4f-4f76-8a0e-a12bcf00835d" (UID: "eed632c4-ac4f-4f76-8a0e-a12bcf00835d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 22:11:41.436534 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:41.436496 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eed632c4-ac4f-4f76-8a0e-a12bcf00835d-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "eed632c4-ac4f-4f76-8a0e-a12bcf00835d" (UID: "eed632c4-ac4f-4f76-8a0e-a12bcf00835d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 22:11:41.438840 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:41.438813 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eed632c4-ac4f-4f76-8a0e-a12bcf00835d-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "eed632c4-ac4f-4f76-8a0e-a12bcf00835d" (UID: "eed632c4-ac4f-4f76-8a0e-a12bcf00835d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:11:41.438840 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:41.438825 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eed632c4-ac4f-4f76-8a0e-a12bcf00835d-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "eed632c4-ac4f-4f76-8a0e-a12bcf00835d" (UID: "eed632c4-ac4f-4f76-8a0e-a12bcf00835d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 22:11:41.439006 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:41.438922 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eed632c4-ac4f-4f76-8a0e-a12bcf00835d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "eed632c4-ac4f-4f76-8a0e-a12bcf00835d" (UID: "eed632c4-ac4f-4f76-8a0e-a12bcf00835d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:11:41.439006 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:41.438942 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eed632c4-ac4f-4f76-8a0e-a12bcf00835d-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "eed632c4-ac4f-4f76-8a0e-a12bcf00835d" (UID: "eed632c4-ac4f-4f76-8a0e-a12bcf00835d"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 22:11:41.439006 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:41.438988 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eed632c4-ac4f-4f76-8a0e-a12bcf00835d-kube-api-access-xgb67" (OuterVolumeSpecName: "kube-api-access-xgb67") pod "eed632c4-ac4f-4f76-8a0e-a12bcf00835d" (UID: "eed632c4-ac4f-4f76-8a0e-a12bcf00835d"). InnerVolumeSpecName "kube-api-access-xgb67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:11:41.445253 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:41.445229 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eed632c4-ac4f-4f76-8a0e-a12bcf00835d-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "eed632c4-ac4f-4f76-8a0e-a12bcf00835d" (UID: "eed632c4-ac4f-4f76-8a0e-a12bcf00835d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:11:41.537553 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:41.537475 2553 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/eed632c4-ac4f-4f76-8a0e-a12bcf00835d-registry-certificates\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:11:41.537553 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:41.537504 2553 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/eed632c4-ac4f-4f76-8a0e-a12bcf00835d-ca-trust-extracted\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:11:41.537553 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:41.537517 2553 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eed632c4-ac4f-4f76-8a0e-a12bcf00835d-bound-sa-token\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:11:41.537553 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:41.537526 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgb67\" (UniqueName: \"kubernetes.io/projected/eed632c4-ac4f-4f76-8a0e-a12bcf00835d-kube-api-access-xgb67\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:11:41.537553 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:41.537535 2553 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/eed632c4-ac4f-4f76-8a0e-a12bcf00835d-installation-pull-secrets\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:11:41.537553 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:41.537545 2553 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/eed632c4-ac4f-4f76-8a0e-a12bcf00835d-image-registry-private-configuration\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:11:41.537553 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:41.537557 2553 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eed632c4-ac4f-4f76-8a0e-a12bcf00835d-registry-tls\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:11:41.537978 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:41.537570 2553 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eed632c4-ac4f-4f76-8a0e-a12bcf00835d-trusted-ca\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:11:42.077907 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:42.077865 2553 generic.go:356] "Generic (PLEG): container finished" podID="eed632c4-ac4f-4f76-8a0e-a12bcf00835d" containerID="1744ef37338915fb8efaa20730d288c9bdbd76ebbb1e8788f4e1d6365b4efc1e" exitCode=0 Jul 28 22:11:42.078343 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:42.077916 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65465d7954-dqhtb" Jul 28 22:11:42.078343 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:42.077949 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65465d7954-dqhtb" event={"ID":"eed632c4-ac4f-4f76-8a0e-a12bcf00835d","Type":"ContainerDied","Data":"1744ef37338915fb8efaa20730d288c9bdbd76ebbb1e8788f4e1d6365b4efc1e"} Jul 28 22:11:42.078343 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:42.078003 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65465d7954-dqhtb" event={"ID":"eed632c4-ac4f-4f76-8a0e-a12bcf00835d","Type":"ContainerDied","Data":"0c56ec0b3d119ece4926952a936522b66bdad6f58ebe2583c66821e141087044"} Jul 28 22:11:42.078343 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:42.078029 2553 scope.go:117] "RemoveContainer" containerID="1744ef37338915fb8efaa20730d288c9bdbd76ebbb1e8788f4e1d6365b4efc1e" Jul 28 22:11:42.086807 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:42.086784 2553 scope.go:117] "RemoveContainer" containerID="1744ef37338915fb8efaa20730d288c9bdbd76ebbb1e8788f4e1d6365b4efc1e" Jul 28 22:11:42.087080 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:11:42.087062 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1744ef37338915fb8efaa20730d288c9bdbd76ebbb1e8788f4e1d6365b4efc1e\": container with ID starting with 1744ef37338915fb8efaa20730d288c9bdbd76ebbb1e8788f4e1d6365b4efc1e not found: ID does not exist" containerID="1744ef37338915fb8efaa20730d288c9bdbd76ebbb1e8788f4e1d6365b4efc1e" Jul 28 22:11:42.087134 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:42.087089 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1744ef37338915fb8efaa20730d288c9bdbd76ebbb1e8788f4e1d6365b4efc1e"} err="failed to get container status \"1744ef37338915fb8efaa20730d288c9bdbd76ebbb1e8788f4e1d6365b4efc1e\": rpc error: code = NotFound desc = could not find container \"1744ef37338915fb8efaa20730d288c9bdbd76ebbb1e8788f4e1d6365b4efc1e\": container with ID starting with 1744ef37338915fb8efaa20730d288c9bdbd76ebbb1e8788f4e1d6365b4efc1e not found: ID does not exist" Jul 28 22:11:42.097972 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:42.097946 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-65465d7954-dqhtb"] Jul 28 22:11:42.101242 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:42.101221 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-65465d7954-dqhtb"] Jul 28 22:11:42.404583 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:42.404498 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eed632c4-ac4f-4f76-8a0e-a12bcf00835d" path="/var/lib/kubelet/pods/eed632c4-ac4f-4f76-8a0e-a12bcf00835d/volumes" Jul 28 22:11:46.095178 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:46.095142 2553 generic.go:356] "Generic (PLEG): container finished" podID="e1d395ef-80a9-4851-ad04-f6377dd12795" containerID="30698e93ae7f59980a2eda0481fdf9d4135e16716211abee531f547bd74fbc88" exitCode=0 Jul 28 22:11:46.095593 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:46.095214 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-7mp2z" event={"ID":"e1d395ef-80a9-4851-ad04-f6377dd12795","Type":"ContainerDied","Data":"30698e93ae7f59980a2eda0481fdf9d4135e16716211abee531f547bd74fbc88"} Jul 28 22:11:46.095633 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:46.095591 2553 scope.go:117] "RemoveContainer" containerID="30698e93ae7f59980a2eda0481fdf9d4135e16716211abee531f547bd74fbc88" Jul 28 22:11:47.100320 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:47.100280 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-7mp2z" event={"ID":"e1d395ef-80a9-4851-ad04-f6377dd12795","Type":"ContainerStarted","Data":"467c8ad25e16734ae85b2e76b4565c6210d26b1837d62ceeb4ed192327b86155"} Jul 28 22:11:47.101767 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:47.101745 2553 generic.go:356] "Generic (PLEG): container finished" podID="414a9396-e46b-433a-9976-95c5c0b1d897" containerID="0b9d4645e52ed06281a88ee6132a90771294621d17f602e623280ba38a5f0920" exitCode=0 Jul 28 22:11:47.101895 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:47.101821 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-bpt84" event={"ID":"414a9396-e46b-433a-9976-95c5c0b1d897","Type":"ContainerDied","Data":"0b9d4645e52ed06281a88ee6132a90771294621d17f602e623280ba38a5f0920"} Jul 28 22:11:47.102162 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:47.102146 2553 scope.go:117] "RemoveContainer" containerID="0b9d4645e52ed06281a88ee6132a90771294621d17f602e623280ba38a5f0920" Jul 28 22:11:48.107845 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:48.107812 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-bpt84" event={"ID":"414a9396-e46b-433a-9976-95c5c0b1d897","Type":"ContainerStarted","Data":"72f11db7fdb2d11ee6f7d797d9728c4d6d49d8f0a8a149f7bd9d3859f4b7d20a"} Jul 28 22:11:56.140643 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:56.140607 2553 generic.go:356] "Generic (PLEG): container finished" podID="c1aff3e5-115b-4502-899a-b2137168973b" containerID="1f27a3dc4bd7d2437c187d6243ac6dedff608beb427b95077bca8e73e249e302" exitCode=0 Jul 28 22:11:56.141134 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:56.140690 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nzswh" event={"ID":"c1aff3e5-115b-4502-899a-b2137168973b","Type":"ContainerDied","Data":"1f27a3dc4bd7d2437c187d6243ac6dedff608beb427b95077bca8e73e249e302"} Jul 28 22:11:56.141203 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:56.141149 2553 scope.go:117] "RemoveContainer" containerID="1f27a3dc4bd7d2437c187d6243ac6dedff608beb427b95077bca8e73e249e302" Jul 28 22:11:57.145738 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:11:57.145693 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nzswh" event={"ID":"c1aff3e5-115b-4502-899a-b2137168973b","Type":"ContainerStarted","Data":"95b642ca3638f53db90b43d64fe1699c549e1beb1d7f29096420ed91597d231e"} Jul 28 22:12:44.928562 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:12:44.928531 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6dd6575cd5-zxwkt"] Jul 28 22:12:44.928973 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:12:44.928833 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eed632c4-ac4f-4f76-8a0e-a12bcf00835d" containerName="registry" Jul 28 22:12:44.928973 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:12:44.928845 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="eed632c4-ac4f-4f76-8a0e-a12bcf00835d" containerName="registry" Jul 28 22:12:44.928973 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:12:44.928903 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="eed632c4-ac4f-4f76-8a0e-a12bcf00835d" containerName="registry" Jul 28 22:12:44.931561 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:12:44.931532 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dd6575cd5-zxwkt" Jul 28 22:12:44.944489 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:12:44.944458 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6dd6575cd5-zxwkt"] Jul 28 22:12:45.041197 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:12:45.041167 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e4e4da99-d77a-4219-b075-d5d3f4715547-console-serving-cert\") pod \"console-6dd6575cd5-zxwkt\" (UID: \"e4e4da99-d77a-4219-b075-d5d3f4715547\") " pod="openshift-console/console-6dd6575cd5-zxwkt" Jul 28 22:12:45.041596 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:12:45.041200 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e4e4da99-d77a-4219-b075-d5d3f4715547-console-oauth-config\") pod \"console-6dd6575cd5-zxwkt\" (UID: \"e4e4da99-d77a-4219-b075-d5d3f4715547\") " pod="openshift-console/console-6dd6575cd5-zxwkt" Jul 28 22:12:45.041860 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:12:45.041831 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e4e4da99-d77a-4219-b075-d5d3f4715547-oauth-serving-cert\") pod \"console-6dd6575cd5-zxwkt\" (UID: \"e4e4da99-d77a-4219-b075-d5d3f4715547\") " pod="openshift-console/console-6dd6575cd5-zxwkt" Jul 28 22:12:45.041957 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:12:45.041898 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4e4da99-d77a-4219-b075-d5d3f4715547-trusted-ca-bundle\") pod \"console-6dd6575cd5-zxwkt\" (UID: \"e4e4da99-d77a-4219-b075-d5d3f4715547\") " pod="openshift-console/console-6dd6575cd5-zxwkt" Jul 28 22:12:45.041957 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:12:45.041940 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e4e4da99-d77a-4219-b075-d5d3f4715547-service-ca\") pod \"console-6dd6575cd5-zxwkt\" (UID: \"e4e4da99-d77a-4219-b075-d5d3f4715547\") " pod="openshift-console/console-6dd6575cd5-zxwkt" Jul 28 22:12:45.042066 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:12:45.041991 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvgt9\" (UniqueName: \"kubernetes.io/projected/e4e4da99-d77a-4219-b075-d5d3f4715547-kube-api-access-tvgt9\") pod \"console-6dd6575cd5-zxwkt\" (UID: \"e4e4da99-d77a-4219-b075-d5d3f4715547\") " pod="openshift-console/console-6dd6575cd5-zxwkt" Jul 28 22:12:45.042066 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:12:45.042054 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e4e4da99-d77a-4219-b075-d5d3f4715547-console-config\") pod \"console-6dd6575cd5-zxwkt\" (UID: \"e4e4da99-d77a-4219-b075-d5d3f4715547\") " pod="openshift-console/console-6dd6575cd5-zxwkt" Jul 28 22:12:45.143174 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:12:45.143131 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e4e4da99-d77a-4219-b075-d5d3f4715547-service-ca\") pod \"console-6dd6575cd5-zxwkt\" (UID: \"e4e4da99-d77a-4219-b075-d5d3f4715547\") " pod="openshift-console/console-6dd6575cd5-zxwkt" Jul 28 22:12:45.143174 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:12:45.143182 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvgt9\" (UniqueName: \"kubernetes.io/projected/e4e4da99-d77a-4219-b075-d5d3f4715547-kube-api-access-tvgt9\") pod \"console-6dd6575cd5-zxwkt\" (UID: \"e4e4da99-d77a-4219-b075-d5d3f4715547\") " pod="openshift-console/console-6dd6575cd5-zxwkt" Jul 28 22:12:45.143408 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:12:45.143220 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e4e4da99-d77a-4219-b075-d5d3f4715547-console-config\") pod \"console-6dd6575cd5-zxwkt\" (UID: \"e4e4da99-d77a-4219-b075-d5d3f4715547\") " pod="openshift-console/console-6dd6575cd5-zxwkt" Jul 28 22:12:45.143408 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:12:45.143252 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e4e4da99-d77a-4219-b075-d5d3f4715547-console-serving-cert\") pod \"console-6dd6575cd5-zxwkt\" (UID: \"e4e4da99-d77a-4219-b075-d5d3f4715547\") " pod="openshift-console/console-6dd6575cd5-zxwkt" Jul 28 22:12:45.143408 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:12:45.143266 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e4e4da99-d77a-4219-b075-d5d3f4715547-console-oauth-config\") pod \"console-6dd6575cd5-zxwkt\" (UID: \"e4e4da99-d77a-4219-b075-d5d3f4715547\") " pod="openshift-console/console-6dd6575cd5-zxwkt" Jul 28 22:12:45.143591 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:12:45.143505 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e4e4da99-d77a-4219-b075-d5d3f4715547-oauth-serving-cert\") pod \"console-6dd6575cd5-zxwkt\" (UID: \"e4e4da99-d77a-4219-b075-d5d3f4715547\") " pod="openshift-console/console-6dd6575cd5-zxwkt" Jul 28 22:12:45.143591 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:12:45.143555 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4e4da99-d77a-4219-b075-d5d3f4715547-trusted-ca-bundle\") pod \"console-6dd6575cd5-zxwkt\" (UID: \"e4e4da99-d77a-4219-b075-d5d3f4715547\") " pod="openshift-console/console-6dd6575cd5-zxwkt" Jul 28 22:12:45.143991 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:12:45.143968 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e4e4da99-d77a-4219-b075-d5d3f4715547-service-ca\") pod \"console-6dd6575cd5-zxwkt\" (UID: \"e4e4da99-d77a-4219-b075-d5d3f4715547\") " pod="openshift-console/console-6dd6575cd5-zxwkt" Jul 28 22:12:45.144229 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:12:45.144206 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e4e4da99-d77a-4219-b075-d5d3f4715547-console-config\") pod \"console-6dd6575cd5-zxwkt\" (UID: \"e4e4da99-d77a-4219-b075-d5d3f4715547\") " pod="openshift-console/console-6dd6575cd5-zxwkt" Jul 28 22:12:45.144333 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:12:45.144316 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4e4da99-d77a-4219-b075-d5d3f4715547-trusted-ca-bundle\") pod \"console-6dd6575cd5-zxwkt\" (UID: \"e4e4da99-d77a-4219-b075-d5d3f4715547\") " pod="openshift-console/console-6dd6575cd5-zxwkt" Jul 28 22:12:45.144428 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:12:45.144406 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e4e4da99-d77a-4219-b075-d5d3f4715547-oauth-serving-cert\") pod \"console-6dd6575cd5-zxwkt\" (UID: \"e4e4da99-d77a-4219-b075-d5d3f4715547\") " pod="openshift-console/console-6dd6575cd5-zxwkt" Jul 28 22:12:45.145961 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:12:45.145939 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e4e4da99-d77a-4219-b075-d5d3f4715547-console-oauth-config\") pod \"console-6dd6575cd5-zxwkt\" (UID: \"e4e4da99-d77a-4219-b075-d5d3f4715547\") " pod="openshift-console/console-6dd6575cd5-zxwkt" Jul 28 22:12:45.146052 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:12:45.145939 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e4e4da99-d77a-4219-b075-d5d3f4715547-console-serving-cert\") pod \"console-6dd6575cd5-zxwkt\" (UID: \"e4e4da99-d77a-4219-b075-d5d3f4715547\") " pod="openshift-console/console-6dd6575cd5-zxwkt" Jul 28 22:12:45.152668 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:12:45.152647 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvgt9\" (UniqueName: \"kubernetes.io/projected/e4e4da99-d77a-4219-b075-d5d3f4715547-kube-api-access-tvgt9\") pod \"console-6dd6575cd5-zxwkt\" (UID: \"e4e4da99-d77a-4219-b075-d5d3f4715547\") " pod="openshift-console/console-6dd6575cd5-zxwkt" Jul 28 22:12:45.242223 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:12:45.242195 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dd6575cd5-zxwkt" Jul 28 22:12:45.371437 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:12:45.371411 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6dd6575cd5-zxwkt"] Jul 28 22:12:45.374006 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:12:45.373981 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4e4da99_d77a_4219_b075_d5d3f4715547.slice/crio-80738b97abac773d691304625561c52520bbcf2680d44ad4d2aedf11347c318e WatchSource:0}: Error finding container 80738b97abac773d691304625561c52520bbcf2680d44ad4d2aedf11347c318e: Status 404 returned error can't find the container with id 80738b97abac773d691304625561c52520bbcf2680d44ad4d2aedf11347c318e Jul 28 22:12:46.334792 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:12:46.334754 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dd6575cd5-zxwkt" event={"ID":"e4e4da99-d77a-4219-b075-d5d3f4715547","Type":"ContainerStarted","Data":"cdcabeedacf7f88bd36d5c42e10bbb5e9b9cf9822aafc4ea1be5619d4520f4bb"} Jul 28 22:12:46.335209 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:12:46.334806 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dd6575cd5-zxwkt" event={"ID":"e4e4da99-d77a-4219-b075-d5d3f4715547","Type":"ContainerStarted","Data":"80738b97abac773d691304625561c52520bbcf2680d44ad4d2aedf11347c318e"} Jul 28 22:12:46.354805 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:12:46.354755 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6dd6575cd5-zxwkt" podStartSLOduration=2.354738695 podStartE2EDuration="2.354738695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:12:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:12:46.353290215 +0000 UTC m=+186.440548303" watchObservedRunningTime="2026-07-28 22:12:46.354738695 +0000 UTC m=+186.441996764" Jul 28 22:12:55.242758 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:12:55.242686 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6dd6575cd5-zxwkt" Jul 28 22:12:55.243394 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:12:55.242812 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6dd6575cd5-zxwkt" Jul 28 22:12:55.247669 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:12:55.247645 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6dd6575cd5-zxwkt" Jul 28 22:12:55.372106 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:12:55.372079 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6dd6575cd5-zxwkt" Jul 28 22:12:55.418298 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:12:55.418266 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d86f68db7-v2w7n"] Jul 28 22:13:20.442055 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:20.442006 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-d86f68db7-v2w7n" podUID="f1169058-aba8-4b2b-9d11-23cb12a61ff1" containerName="console" containerID="cri-o://6528fd0fd515fe87256262c3ede663d5882afc97510fa86c849ce5b95f8b097a" gracePeriod=15 Jul 28 22:13:20.679115 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:20.679090 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d86f68db7-v2w7n" Jul 28 22:13:20.832844 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:20.832811 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f1169058-aba8-4b2b-9d11-23cb12a61ff1-console-config\") pod \"f1169058-aba8-4b2b-9d11-23cb12a61ff1\" (UID: \"f1169058-aba8-4b2b-9d11-23cb12a61ff1\") " Jul 28 22:13:20.833033 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:20.832860 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f1169058-aba8-4b2b-9d11-23cb12a61ff1-console-serving-cert\") pod \"f1169058-aba8-4b2b-9d11-23cb12a61ff1\" (UID: \"f1169058-aba8-4b2b-9d11-23cb12a61ff1\") " Jul 28 22:13:20.833033 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:20.832885 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f1169058-aba8-4b2b-9d11-23cb12a61ff1-console-oauth-config\") pod \"f1169058-aba8-4b2b-9d11-23cb12a61ff1\" (UID: \"f1169058-aba8-4b2b-9d11-23cb12a61ff1\") " Jul 28 22:13:20.833033 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:20.832911 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f1169058-aba8-4b2b-9d11-23cb12a61ff1-service-ca\") pod \"f1169058-aba8-4b2b-9d11-23cb12a61ff1\" (UID: \"f1169058-aba8-4b2b-9d11-23cb12a61ff1\") " Jul 28 22:13:20.833033 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:20.832939 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1169058-aba8-4b2b-9d11-23cb12a61ff1-trusted-ca-bundle\") pod \"f1169058-aba8-4b2b-9d11-23cb12a61ff1\" (UID: \"f1169058-aba8-4b2b-9d11-23cb12a61ff1\") " Jul 28 22:13:20.833033 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:20.832971 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f1169058-aba8-4b2b-9d11-23cb12a61ff1-oauth-serving-cert\") pod \"f1169058-aba8-4b2b-9d11-23cb12a61ff1\" (UID: \"f1169058-aba8-4b2b-9d11-23cb12a61ff1\") " Jul 28 22:13:20.833033 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:20.832996 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cj26c\" (UniqueName: \"kubernetes.io/projected/f1169058-aba8-4b2b-9d11-23cb12a61ff1-kube-api-access-cj26c\") pod \"f1169058-aba8-4b2b-9d11-23cb12a61ff1\" (UID: \"f1169058-aba8-4b2b-9d11-23cb12a61ff1\") " Jul 28 22:13:20.833364 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:20.833336 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1169058-aba8-4b2b-9d11-23cb12a61ff1-console-config" (OuterVolumeSpecName: "console-config") pod "f1169058-aba8-4b2b-9d11-23cb12a61ff1" (UID: "f1169058-aba8-4b2b-9d11-23cb12a61ff1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 22:13:20.833461 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:20.833430 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1169058-aba8-4b2b-9d11-23cb12a61ff1-service-ca" (OuterVolumeSpecName: "service-ca") pod "f1169058-aba8-4b2b-9d11-23cb12a61ff1" (UID: "f1169058-aba8-4b2b-9d11-23cb12a61ff1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 22:13:20.833461 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:20.833451 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1169058-aba8-4b2b-9d11-23cb12a61ff1-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f1169058-aba8-4b2b-9d11-23cb12a61ff1" (UID: "f1169058-aba8-4b2b-9d11-23cb12a61ff1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 22:13:20.833610 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:20.833433 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1169058-aba8-4b2b-9d11-23cb12a61ff1-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "f1169058-aba8-4b2b-9d11-23cb12a61ff1" (UID: "f1169058-aba8-4b2b-9d11-23cb12a61ff1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 22:13:20.835289 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:20.835265 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1169058-aba8-4b2b-9d11-23cb12a61ff1-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f1169058-aba8-4b2b-9d11-23cb12a61ff1" (UID: "f1169058-aba8-4b2b-9d11-23cb12a61ff1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 22:13:20.835368 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:20.835340 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1169058-aba8-4b2b-9d11-23cb12a61ff1-kube-api-access-cj26c" (OuterVolumeSpecName: "kube-api-access-cj26c") pod "f1169058-aba8-4b2b-9d11-23cb12a61ff1" (UID: "f1169058-aba8-4b2b-9d11-23cb12a61ff1"). InnerVolumeSpecName "kube-api-access-cj26c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:13:20.835414 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:20.835383 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1169058-aba8-4b2b-9d11-23cb12a61ff1-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f1169058-aba8-4b2b-9d11-23cb12a61ff1" (UID: "f1169058-aba8-4b2b-9d11-23cb12a61ff1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 22:13:20.934096 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:20.934058 2553 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f1169058-aba8-4b2b-9d11-23cb12a61ff1-console-oauth-config\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:13:20.934096 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:20.934088 2553 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f1169058-aba8-4b2b-9d11-23cb12a61ff1-service-ca\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:13:20.934096 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:20.934098 2553 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1169058-aba8-4b2b-9d11-23cb12a61ff1-trusted-ca-bundle\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:13:20.934096 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:20.934107 2553 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f1169058-aba8-4b2b-9d11-23cb12a61ff1-oauth-serving-cert\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:13:20.934354 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:20.934115 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj26c\" (UniqueName: \"kubernetes.io/projected/f1169058-aba8-4b2b-9d11-23cb12a61ff1-kube-api-access-cj26c\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:13:20.934354 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:20.934123 2553 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f1169058-aba8-4b2b-9d11-23cb12a61ff1-console-config\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:13:20.934354 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:20.934131 2553 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f1169058-aba8-4b2b-9d11-23cb12a61ff1-console-serving-cert\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:13:21.468563 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:21.468528 2553 generic.go:356] "Generic (PLEG): container finished" podID="f1169058-aba8-4b2b-9d11-23cb12a61ff1" containerID="6528fd0fd515fe87256262c3ede663d5882afc97510fa86c849ce5b95f8b097a" exitCode=0 Jul 28 22:13:21.469013 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:21.468576 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d86f68db7-v2w7n" Jul 28 22:13:21.469013 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:21.468590 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d86f68db7-v2w7n" event={"ID":"f1169058-aba8-4b2b-9d11-23cb12a61ff1","Type":"ContainerDied","Data":"6528fd0fd515fe87256262c3ede663d5882afc97510fa86c849ce5b95f8b097a"} Jul 28 22:13:21.469013 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:21.468641 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d86f68db7-v2w7n" event={"ID":"f1169058-aba8-4b2b-9d11-23cb12a61ff1","Type":"ContainerDied","Data":"fa493fc6b75bdb011b90e264c1ce1eefdf460ffb638360258cc2f012e5c441f9"} Jul 28 22:13:21.469013 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:21.468660 2553 scope.go:117] "RemoveContainer" containerID="6528fd0fd515fe87256262c3ede663d5882afc97510fa86c849ce5b95f8b097a" Jul 28 22:13:21.477212 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:21.477196 2553 scope.go:117] "RemoveContainer" containerID="6528fd0fd515fe87256262c3ede663d5882afc97510fa86c849ce5b95f8b097a" Jul 28 22:13:21.477447 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:13:21.477430 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6528fd0fd515fe87256262c3ede663d5882afc97510fa86c849ce5b95f8b097a\": container with ID starting with 6528fd0fd515fe87256262c3ede663d5882afc97510fa86c849ce5b95f8b097a not found: ID does not exist" containerID="6528fd0fd515fe87256262c3ede663d5882afc97510fa86c849ce5b95f8b097a" Jul 28 22:13:21.477493 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:21.477453 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6528fd0fd515fe87256262c3ede663d5882afc97510fa86c849ce5b95f8b097a"} err="failed to get container status \"6528fd0fd515fe87256262c3ede663d5882afc97510fa86c849ce5b95f8b097a\": rpc error: code = NotFound desc = could not find container \"6528fd0fd515fe87256262c3ede663d5882afc97510fa86c849ce5b95f8b097a\": container with ID starting with 6528fd0fd515fe87256262c3ede663d5882afc97510fa86c849ce5b95f8b097a not found: ID does not exist" Jul 28 22:13:21.489206 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:21.489172 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d86f68db7-v2w7n"] Jul 28 22:13:21.493476 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:21.493457 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-d86f68db7-v2w7n"] Jul 28 22:13:22.404078 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:22.404047 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1169058-aba8-4b2b-9d11-23cb12a61ff1" path="/var/lib/kubelet/pods/f1169058-aba8-4b2b-9d11-23cb12a61ff1/volumes" Jul 28 22:13:26.815676 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:26.815639 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nz5xt"] Jul 28 22:13:26.816167 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:26.815925 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1169058-aba8-4b2b-9d11-23cb12a61ff1" containerName="console" Jul 28 22:13:26.816167 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:26.815936 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1169058-aba8-4b2b-9d11-23cb12a61ff1" containerName="console" Jul 28 22:13:26.816167 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:26.815996 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1169058-aba8-4b2b-9d11-23cb12a61ff1" containerName="console" Jul 28 22:13:26.818341 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:26.818325 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nz5xt" Jul 28 22:13:26.820767 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:26.820744 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 28 22:13:26.821332 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:26.821318 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-dclqn\"" Jul 28 22:13:26.821426 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:26.821323 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 28 22:13:26.827670 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:26.827644 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nz5xt"] Jul 28 22:13:26.883275 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:26.883239 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c75e4b8b-a595-4e90-99f2-50b0d2884c0c-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nz5xt\" (UID: \"c75e4b8b-a595-4e90-99f2-50b0d2884c0c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nz5xt" Jul 28 22:13:26.883432 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:26.883282 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c75e4b8b-a595-4e90-99f2-50b0d2884c0c-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nz5xt\" (UID: \"c75e4b8b-a595-4e90-99f2-50b0d2884c0c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nz5xt" Jul 28 22:13:26.883432 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:26.883305 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5tkq\" (UniqueName: \"kubernetes.io/projected/c75e4b8b-a595-4e90-99f2-50b0d2884c0c-kube-api-access-n5tkq\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nz5xt\" (UID: \"c75e4b8b-a595-4e90-99f2-50b0d2884c0c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nz5xt" Jul 28 22:13:26.983916 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:26.983884 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c75e4b8b-a595-4e90-99f2-50b0d2884c0c-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nz5xt\" (UID: \"c75e4b8b-a595-4e90-99f2-50b0d2884c0c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nz5xt" Jul 28 22:13:26.983916 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:26.983920 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c75e4b8b-a595-4e90-99f2-50b0d2884c0c-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nz5xt\" (UID: \"c75e4b8b-a595-4e90-99f2-50b0d2884c0c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nz5xt" Jul 28 22:13:26.984154 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:26.983944 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5tkq\" (UniqueName: \"kubernetes.io/projected/c75e4b8b-a595-4e90-99f2-50b0d2884c0c-kube-api-access-n5tkq\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nz5xt\" (UID: \"c75e4b8b-a595-4e90-99f2-50b0d2884c0c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nz5xt" Jul 28 22:13:26.984267 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:26.984245 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c75e4b8b-a595-4e90-99f2-50b0d2884c0c-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nz5xt\" (UID: \"c75e4b8b-a595-4e90-99f2-50b0d2884c0c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nz5xt" Jul 28 22:13:26.984354 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:26.984329 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c75e4b8b-a595-4e90-99f2-50b0d2884c0c-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nz5xt\" (UID: \"c75e4b8b-a595-4e90-99f2-50b0d2884c0c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nz5xt" Jul 28 22:13:26.992233 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:26.992203 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5tkq\" (UniqueName: \"kubernetes.io/projected/c75e4b8b-a595-4e90-99f2-50b0d2884c0c-kube-api-access-n5tkq\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nz5xt\" (UID: \"c75e4b8b-a595-4e90-99f2-50b0d2884c0c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nz5xt" Jul 28 22:13:27.128106 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:27.128020 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nz5xt" Jul 28 22:13:27.254158 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:27.254133 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nz5xt"] Jul 28 22:13:27.256914 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:13:27.256891 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc75e4b8b_a595_4e90_99f2_50b0d2884c0c.slice/crio-2fe23d5fe7fa81832168e7130a5770965edcd746954733144c3718972042afdf WatchSource:0}: Error finding container 2fe23d5fe7fa81832168e7130a5770965edcd746954733144c3718972042afdf: Status 404 returned error can't find the container with id 2fe23d5fe7fa81832168e7130a5770965edcd746954733144c3718972042afdf Jul 28 22:13:27.493253 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:27.493224 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nz5xt" event={"ID":"c75e4b8b-a595-4e90-99f2-50b0d2884c0c","Type":"ContainerStarted","Data":"2fe23d5fe7fa81832168e7130a5770965edcd746954733144c3718972042afdf"} Jul 28 22:13:34.526502 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:34.526469 2553 generic.go:356] "Generic (PLEG): container finished" podID="c75e4b8b-a595-4e90-99f2-50b0d2884c0c" containerID="7aa89c1977c4135b48548d8906323a2ffdda855bfa7a8195137bfd238a6a0d0a" exitCode=0 Jul 28 22:13:34.527039 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:34.526533 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nz5xt" event={"ID":"c75e4b8b-a595-4e90-99f2-50b0d2884c0c","Type":"ContainerDied","Data":"7aa89c1977c4135b48548d8906323a2ffdda855bfa7a8195137bfd238a6a0d0a"} Jul 28 22:13:37.540610 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:37.540577 2553 generic.go:356] "Generic (PLEG): container finished" podID="c75e4b8b-a595-4e90-99f2-50b0d2884c0c" containerID="077d40a92ea9352fa59d01043ff3e70ec3eceea0089fadbe99ade60ab9f8091f" exitCode=0 Jul 28 22:13:37.541005 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:37.540655 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nz5xt" event={"ID":"c75e4b8b-a595-4e90-99f2-50b0d2884c0c","Type":"ContainerDied","Data":"077d40a92ea9352fa59d01043ff3e70ec3eceea0089fadbe99ade60ab9f8091f"} Jul 28 22:13:44.571463 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:44.571423 2553 generic.go:356] "Generic (PLEG): container finished" podID="c75e4b8b-a595-4e90-99f2-50b0d2884c0c" containerID="0bb805e043e550d9bb0ab8c19ab2edc40b88fccdacc8397d6072551f116730c7" exitCode=0 Jul 28 22:13:44.571918 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:44.571503 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nz5xt" event={"ID":"c75e4b8b-a595-4e90-99f2-50b0d2884c0c","Type":"ContainerDied","Data":"0bb805e043e550d9bb0ab8c19ab2edc40b88fccdacc8397d6072551f116730c7"} Jul 28 22:13:45.700873 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:45.700851 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nz5xt" Jul 28 22:13:45.746054 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:45.746029 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n5tkq\" (UniqueName: \"kubernetes.io/projected/c75e4b8b-a595-4e90-99f2-50b0d2884c0c-kube-api-access-n5tkq\") pod \"c75e4b8b-a595-4e90-99f2-50b0d2884c0c\" (UID: \"c75e4b8b-a595-4e90-99f2-50b0d2884c0c\") " Jul 28 22:13:45.746212 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:45.746082 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c75e4b8b-a595-4e90-99f2-50b0d2884c0c-bundle\") pod \"c75e4b8b-a595-4e90-99f2-50b0d2884c0c\" (UID: \"c75e4b8b-a595-4e90-99f2-50b0d2884c0c\") " Jul 28 22:13:45.746212 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:45.746161 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c75e4b8b-a595-4e90-99f2-50b0d2884c0c-util\") pod \"c75e4b8b-a595-4e90-99f2-50b0d2884c0c\" (UID: \"c75e4b8b-a595-4e90-99f2-50b0d2884c0c\") " Jul 28 22:13:45.746823 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:45.746761 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c75e4b8b-a595-4e90-99f2-50b0d2884c0c-bundle" (OuterVolumeSpecName: "bundle") pod "c75e4b8b-a595-4e90-99f2-50b0d2884c0c" (UID: "c75e4b8b-a595-4e90-99f2-50b0d2884c0c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:13:45.748357 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:45.748330 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c75e4b8b-a595-4e90-99f2-50b0d2884c0c-kube-api-access-n5tkq" (OuterVolumeSpecName: "kube-api-access-n5tkq") pod "c75e4b8b-a595-4e90-99f2-50b0d2884c0c" (UID: "c75e4b8b-a595-4e90-99f2-50b0d2884c0c"). InnerVolumeSpecName "kube-api-access-n5tkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:13:45.750263 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:45.750240 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c75e4b8b-a595-4e90-99f2-50b0d2884c0c-util" (OuterVolumeSpecName: "util") pod "c75e4b8b-a595-4e90-99f2-50b0d2884c0c" (UID: "c75e4b8b-a595-4e90-99f2-50b0d2884c0c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:13:45.847012 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:45.846942 2553 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c75e4b8b-a595-4e90-99f2-50b0d2884c0c-util\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:13:45.847012 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:45.846967 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5tkq\" (UniqueName: \"kubernetes.io/projected/c75e4b8b-a595-4e90-99f2-50b0d2884c0c-kube-api-access-n5tkq\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:13:45.847012 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:45.846977 2553 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c75e4b8b-a595-4e90-99f2-50b0d2884c0c-bundle\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:13:46.579651 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:46.579619 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nz5xt" event={"ID":"c75e4b8b-a595-4e90-99f2-50b0d2884c0c","Type":"ContainerDied","Data":"2fe23d5fe7fa81832168e7130a5770965edcd746954733144c3718972042afdf"} Jul 28 22:13:46.579859 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:46.579662 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2fe23d5fe7fa81832168e7130a5770965edcd746954733144c3718972042afdf" Jul 28 22:13:46.579859 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:46.579674 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nz5xt" Jul 28 22:13:49.775059 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:49.775022 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-fkm7v"] Jul 28 22:13:49.775519 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:49.775325 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c75e4b8b-a595-4e90-99f2-50b0d2884c0c" containerName="util" Jul 28 22:13:49.775519 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:49.775336 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="c75e4b8b-a595-4e90-99f2-50b0d2884c0c" containerName="util" Jul 28 22:13:49.775519 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:49.775344 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c75e4b8b-a595-4e90-99f2-50b0d2884c0c" containerName="extract" Jul 28 22:13:49.775519 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:49.775349 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="c75e4b8b-a595-4e90-99f2-50b0d2884c0c" containerName="extract" Jul 28 22:13:49.775519 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:49.775361 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c75e4b8b-a595-4e90-99f2-50b0d2884c0c" containerName="pull" Jul 28 22:13:49.775519 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:49.775366 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="c75e4b8b-a595-4e90-99f2-50b0d2884c0c" containerName="pull" Jul 28 22:13:49.775519 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:49.775411 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="c75e4b8b-a595-4e90-99f2-50b0d2884c0c" containerName="extract" Jul 28 22:13:49.827512 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:49.827479 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-fkm7v"] Jul 28 22:13:49.827664 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:49.827576 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-fkm7v" Jul 28 22:13:49.830719 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:49.830682 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 28 22:13:49.830844 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:49.830721 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 28 22:13:49.830844 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:49.830736 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-8cktt\"" Jul 28 22:13:49.873927 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:49.873894 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md99t\" (UniqueName: \"kubernetes.io/projected/0ec80eeb-85f9-4768-ae0b-9ec0bfde2f60-kube-api-access-md99t\") pod \"cert-manager-operator-controller-manager-55c444bf86-fkm7v\" (UID: \"0ec80eeb-85f9-4768-ae0b-9ec0bfde2f60\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-fkm7v" Jul 28 22:13:49.873927 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:49.873927 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ec80eeb-85f9-4768-ae0b-9ec0bfde2f60-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-fkm7v\" (UID: \"0ec80eeb-85f9-4768-ae0b-9ec0bfde2f60\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-fkm7v" Jul 28 22:13:49.974300 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:49.974261 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md99t\" (UniqueName: \"kubernetes.io/projected/0ec80eeb-85f9-4768-ae0b-9ec0bfde2f60-kube-api-access-md99t\") pod \"cert-manager-operator-controller-manager-55c444bf86-fkm7v\" (UID: \"0ec80eeb-85f9-4768-ae0b-9ec0bfde2f60\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-fkm7v" Jul 28 22:13:49.974300 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:49.974301 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ec80eeb-85f9-4768-ae0b-9ec0bfde2f60-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-fkm7v\" (UID: \"0ec80eeb-85f9-4768-ae0b-9ec0bfde2f60\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-fkm7v" Jul 28 22:13:49.974683 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:49.974662 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ec80eeb-85f9-4768-ae0b-9ec0bfde2f60-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-fkm7v\" (UID: \"0ec80eeb-85f9-4768-ae0b-9ec0bfde2f60\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-fkm7v" Jul 28 22:13:49.983223 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:49.983175 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-md99t\" (UniqueName: \"kubernetes.io/projected/0ec80eeb-85f9-4768-ae0b-9ec0bfde2f60-kube-api-access-md99t\") pod \"cert-manager-operator-controller-manager-55c444bf86-fkm7v\" (UID: \"0ec80eeb-85f9-4768-ae0b-9ec0bfde2f60\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-fkm7v" Jul 28 22:13:50.136494 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:50.136412 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-fkm7v" Jul 28 22:13:50.297382 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:50.297352 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-fkm7v"] Jul 28 22:13:50.300058 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:13:50.300036 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ec80eeb_85f9_4768_ae0b_9ec0bfde2f60.slice/crio-a021776a61b1e15d394e4d98283c59742baf8ec9a6fd61fa7a82ded39768373a WatchSource:0}: Error finding container a021776a61b1e15d394e4d98283c59742baf8ec9a6fd61fa7a82ded39768373a: Status 404 returned error can't find the container with id a021776a61b1e15d394e4d98283c59742baf8ec9a6fd61fa7a82ded39768373a Jul 28 22:13:50.596099 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:50.596065 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-fkm7v" event={"ID":"0ec80eeb-85f9-4768-ae0b-9ec0bfde2f60","Type":"ContainerStarted","Data":"a021776a61b1e15d394e4d98283c59742baf8ec9a6fd61fa7a82ded39768373a"} Jul 28 22:13:53.612234 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:53.612188 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-fkm7v" event={"ID":"0ec80eeb-85f9-4768-ae0b-9ec0bfde2f60","Type":"ContainerStarted","Data":"e27238b54ada644b98156fdd0e41761a0c4f50e0f74440486130c1853fede52a"} Jul 28 22:13:53.638306 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:53.638249 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-fkm7v" podStartSLOduration=1.941165893 podStartE2EDuration="4.6382321s" totalImagesPullingTime="2.697066207s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:13:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:13:53.636134737 +0000 UTC m=+253.723392806" watchObservedRunningTime="2026-07-28 22:13:53.6382321 +0000 UTC m=+253.725490169" Jul 28 22:13:54.519717 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:54.519682 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftwnt6"] Jul 28 22:13:54.522753 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:54.522599 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftwnt6" Jul 28 22:13:54.526383 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:54.526356 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-dclqn\"" Jul 28 22:13:54.526921 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:54.526903 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 28 22:13:54.526989 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:54.526927 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 28 22:13:54.530576 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:54.530555 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftwnt6"] Jul 28 22:13:54.604632 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:54.604595 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/df19a89f-7131-4247-96f0-820e7c63d002-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftwnt6\" (UID: \"df19a89f-7131-4247-96f0-820e7c63d002\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftwnt6" Jul 28 22:13:54.604837 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:54.604655 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/df19a89f-7131-4247-96f0-820e7c63d002-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftwnt6\" (UID: \"df19a89f-7131-4247-96f0-820e7c63d002\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftwnt6" Jul 28 22:13:54.604837 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:54.604697 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6wq4\" (UniqueName: \"kubernetes.io/projected/df19a89f-7131-4247-96f0-820e7c63d002-kube-api-access-w6wq4\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftwnt6\" (UID: \"df19a89f-7131-4247-96f0-820e7c63d002\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftwnt6" Jul 28 22:13:54.705009 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:54.704975 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/df19a89f-7131-4247-96f0-820e7c63d002-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftwnt6\" (UID: \"df19a89f-7131-4247-96f0-820e7c63d002\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftwnt6" Jul 28 22:13:54.705009 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:54.705014 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6wq4\" (UniqueName: \"kubernetes.io/projected/df19a89f-7131-4247-96f0-820e7c63d002-kube-api-access-w6wq4\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftwnt6\" (UID: \"df19a89f-7131-4247-96f0-820e7c63d002\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftwnt6" Jul 28 22:13:54.705533 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:54.705064 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/df19a89f-7131-4247-96f0-820e7c63d002-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftwnt6\" (UID: \"df19a89f-7131-4247-96f0-820e7c63d002\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftwnt6" Jul 28 22:13:54.705533 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:54.705375 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/df19a89f-7131-4247-96f0-820e7c63d002-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftwnt6\" (UID: \"df19a89f-7131-4247-96f0-820e7c63d002\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftwnt6" Jul 28 22:13:54.705533 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:54.705364 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/df19a89f-7131-4247-96f0-820e7c63d002-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftwnt6\" (UID: \"df19a89f-7131-4247-96f0-820e7c63d002\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftwnt6" Jul 28 22:13:54.713980 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:54.713954 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6wq4\" (UniqueName: \"kubernetes.io/projected/df19a89f-7131-4247-96f0-820e7c63d002-kube-api-access-w6wq4\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftwnt6\" (UID: \"df19a89f-7131-4247-96f0-820e7c63d002\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftwnt6" Jul 28 22:13:54.836878 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:54.836795 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftwnt6" Jul 28 22:13:54.971366 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:54.971335 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftwnt6"] Jul 28 22:13:54.974445 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:13:54.974423 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf19a89f_7131_4247_96f0_820e7c63d002.slice/crio-f056c64adfa376d864fea68545f305e33eb2c99cddb81ba8eb94726eaf272243 WatchSource:0}: Error finding container f056c64adfa376d864fea68545f305e33eb2c99cddb81ba8eb94726eaf272243: Status 404 returned error can't find the container with id f056c64adfa376d864fea68545f305e33eb2c99cddb81ba8eb94726eaf272243 Jul 28 22:13:55.622504 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:55.622476 2553 generic.go:356] "Generic (PLEG): container finished" podID="df19a89f-7131-4247-96f0-820e7c63d002" containerID="685fe149ebbd63c7757091b1e7c2d8bbc414c0bd688fb4ecf2e8663ff9801d33" exitCode=0 Jul 28 22:13:55.622662 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:55.622565 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftwnt6" event={"ID":"df19a89f-7131-4247-96f0-820e7c63d002","Type":"ContainerDied","Data":"685fe149ebbd63c7757091b1e7c2d8bbc414c0bd688fb4ecf2e8663ff9801d33"} Jul 28 22:13:55.622662 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:55.622620 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftwnt6" event={"ID":"df19a89f-7131-4247-96f0-820e7c63d002","Type":"ContainerStarted","Data":"f056c64adfa376d864fea68545f305e33eb2c99cddb81ba8eb94726eaf272243"} Jul 28 22:13:57.634157 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:57.634130 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftwnt6" event={"ID":"df19a89f-7131-4247-96f0-820e7c63d002","Type":"ContainerStarted","Data":"fc214c8990bae3e54d6dfc85a7f28b96cf91456d7c9cf5fe0b662e8c7d0cb8cf"} Jul 28 22:13:58.639954 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:58.639923 2553 generic.go:356] "Generic (PLEG): container finished" podID="df19a89f-7131-4247-96f0-820e7c63d002" containerID="fc214c8990bae3e54d6dfc85a7f28b96cf91456d7c9cf5fe0b662e8c7d0cb8cf" exitCode=0 Jul 28 22:13:58.640353 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:58.640015 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftwnt6" event={"ID":"df19a89f-7131-4247-96f0-820e7c63d002","Type":"ContainerDied","Data":"fc214c8990bae3e54d6dfc85a7f28b96cf91456d7c9cf5fe0b662e8c7d0cb8cf"} Jul 28 22:13:59.646012 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:59.645982 2553 generic.go:356] "Generic (PLEG): container finished" podID="df19a89f-7131-4247-96f0-820e7c63d002" containerID="7844f0351231c8cea8198cce4a8b11bd985018875c56aea4a6e804ce207af22b" exitCode=0 Jul 28 22:13:59.646382 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:13:59.646071 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftwnt6" event={"ID":"df19a89f-7131-4247-96f0-820e7c63d002","Type":"ContainerDied","Data":"7844f0351231c8cea8198cce4a8b11bd985018875c56aea4a6e804ce207af22b"} Jul 28 22:14:00.777363 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:00.777336 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftwnt6" Jul 28 22:14:00.848922 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:00.848890 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/df19a89f-7131-4247-96f0-820e7c63d002-util\") pod \"df19a89f-7131-4247-96f0-820e7c63d002\" (UID: \"df19a89f-7131-4247-96f0-820e7c63d002\") " Jul 28 22:14:00.848922 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:00.848924 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w6wq4\" (UniqueName: \"kubernetes.io/projected/df19a89f-7131-4247-96f0-820e7c63d002-kube-api-access-w6wq4\") pod \"df19a89f-7131-4247-96f0-820e7c63d002\" (UID: \"df19a89f-7131-4247-96f0-820e7c63d002\") " Jul 28 22:14:00.849117 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:00.848949 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/df19a89f-7131-4247-96f0-820e7c63d002-bundle\") pod \"df19a89f-7131-4247-96f0-820e7c63d002\" (UID: \"df19a89f-7131-4247-96f0-820e7c63d002\") " Jul 28 22:14:00.849385 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:00.849356 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df19a89f-7131-4247-96f0-820e7c63d002-bundle" (OuterVolumeSpecName: "bundle") pod "df19a89f-7131-4247-96f0-820e7c63d002" (UID: "df19a89f-7131-4247-96f0-820e7c63d002"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:14:00.851228 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:00.851198 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df19a89f-7131-4247-96f0-820e7c63d002-kube-api-access-w6wq4" (OuterVolumeSpecName: "kube-api-access-w6wq4") pod "df19a89f-7131-4247-96f0-820e7c63d002" (UID: "df19a89f-7131-4247-96f0-820e7c63d002"). InnerVolumeSpecName "kube-api-access-w6wq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:14:00.853551 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:00.853532 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df19a89f-7131-4247-96f0-820e7c63d002-util" (OuterVolumeSpecName: "util") pod "df19a89f-7131-4247-96f0-820e7c63d002" (UID: "df19a89f-7131-4247-96f0-820e7c63d002"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:14:00.949528 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:00.949457 2553 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/df19a89f-7131-4247-96f0-820e7c63d002-bundle\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:14:00.949528 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:00.949481 2553 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/df19a89f-7131-4247-96f0-820e7c63d002-util\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:14:00.949528 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:00.949490 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6wq4\" (UniqueName: \"kubernetes.io/projected/df19a89f-7131-4247-96f0-820e7c63d002-kube-api-access-w6wq4\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:14:01.655904 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:01.655871 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftwnt6" event={"ID":"df19a89f-7131-4247-96f0-820e7c63d002","Type":"ContainerDied","Data":"f056c64adfa376d864fea68545f305e33eb2c99cddb81ba8eb94726eaf272243"} Jul 28 22:14:01.656099 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:01.655920 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f056c64adfa376d864fea68545f305e33eb2c99cddb81ba8eb94726eaf272243" Jul 28 22:14:01.656099 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:01.655931 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftwnt6" Jul 28 22:14:01.857367 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:01.857335 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-6tcc5"] Jul 28 22:14:01.857747 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:01.857601 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df19a89f-7131-4247-96f0-820e7c63d002" containerName="extract" Jul 28 22:14:01.857747 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:01.857612 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="df19a89f-7131-4247-96f0-820e7c63d002" containerName="extract" Jul 28 22:14:01.857747 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:01.857627 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df19a89f-7131-4247-96f0-820e7c63d002" containerName="util" Jul 28 22:14:01.857747 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:01.857633 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="df19a89f-7131-4247-96f0-820e7c63d002" containerName="util" Jul 28 22:14:01.857747 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:01.857647 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df19a89f-7131-4247-96f0-820e7c63d002" containerName="pull" Jul 28 22:14:01.857747 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:01.857652 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="df19a89f-7131-4247-96f0-820e7c63d002" containerName="pull" Jul 28 22:14:01.857747 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:01.857715 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="df19a89f-7131-4247-96f0-820e7c63d002" containerName="extract" Jul 28 22:14:01.860591 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:01.860570 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-6tcc5" Jul 28 22:14:01.862990 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:01.862961 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-kgnjf\"" Jul 28 22:14:01.863094 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:01.862990 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 28 22:14:01.863094 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:01.862991 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 28 22:14:01.871827 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:01.871806 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-6tcc5"] Jul 28 22:14:01.957429 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:01.957340 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7b0b78a4-e337-47f6-b757-d5e39eb7d7f2-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-6tcc5\" (UID: \"7b0b78a4-e337-47f6-b757-d5e39eb7d7f2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6tcc5" Jul 28 22:14:01.957429 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:01.957378 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-php77\" (UniqueName: \"kubernetes.io/projected/7b0b78a4-e337-47f6-b757-d5e39eb7d7f2-kube-api-access-php77\") pod \"cert-manager-webhook-6486ffd997-6tcc5\" (UID: \"7b0b78a4-e337-47f6-b757-d5e39eb7d7f2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6tcc5" Jul 28 22:14:02.058003 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:02.057969 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7b0b78a4-e337-47f6-b757-d5e39eb7d7f2-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-6tcc5\" (UID: \"7b0b78a4-e337-47f6-b757-d5e39eb7d7f2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6tcc5" Jul 28 22:14:02.058003 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:02.058007 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-php77\" (UniqueName: \"kubernetes.io/projected/7b0b78a4-e337-47f6-b757-d5e39eb7d7f2-kube-api-access-php77\") pod \"cert-manager-webhook-6486ffd997-6tcc5\" (UID: \"7b0b78a4-e337-47f6-b757-d5e39eb7d7f2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6tcc5" Jul 28 22:14:02.067035 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:02.066999 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7b0b78a4-e337-47f6-b757-d5e39eb7d7f2-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-6tcc5\" (UID: \"7b0b78a4-e337-47f6-b757-d5e39eb7d7f2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6tcc5" Jul 28 22:14:02.067163 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:02.067108 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-php77\" (UniqueName: \"kubernetes.io/projected/7b0b78a4-e337-47f6-b757-d5e39eb7d7f2-kube-api-access-php77\") pod \"cert-manager-webhook-6486ffd997-6tcc5\" (UID: \"7b0b78a4-e337-47f6-b757-d5e39eb7d7f2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6tcc5" Jul 28 22:14:02.183683 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:02.183646 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-6tcc5" Jul 28 22:14:02.313846 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:02.313165 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-6tcc5"] Jul 28 22:14:02.317315 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:14:02.317277 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b0b78a4_e337_47f6_b757_d5e39eb7d7f2.slice/crio-60480c351d556a736c276b2d3b335ade99645186ffcce1ca3ce0c5dde9e65c08 WatchSource:0}: Error finding container 60480c351d556a736c276b2d3b335ade99645186ffcce1ca3ce0c5dde9e65c08: Status 404 returned error can't find the container with id 60480c351d556a736c276b2d3b335ade99645186ffcce1ca3ce0c5dde9e65c08 Jul 28 22:14:02.661796 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:02.661688 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-6tcc5" event={"ID":"7b0b78a4-e337-47f6-b757-d5e39eb7d7f2","Type":"ContainerStarted","Data":"60480c351d556a736c276b2d3b335ade99645186ffcce1ca3ce0c5dde9e65c08"} Jul 28 22:14:04.935130 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:04.935093 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-bjh2h"] Jul 28 22:14:04.938640 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:04.938613 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-bjh2h" Jul 28 22:14:04.940946 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:04.940926 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-8cnbn\"" Jul 28 22:14:04.949997 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:04.949961 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-bjh2h"] Jul 28 22:14:04.980575 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:04.980534 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e4450f6a-62e2-4f76-a3d1-4a8a601b2a2f-bound-sa-token\") pod \"cert-manager-55b4d954d5-bjh2h\" (UID: \"e4450f6a-62e2-4f76-a3d1-4a8a601b2a2f\") " pod="cert-manager/cert-manager-55b4d954d5-bjh2h" Jul 28 22:14:04.980749 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:04.980595 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2lnh\" (UniqueName: \"kubernetes.io/projected/e4450f6a-62e2-4f76-a3d1-4a8a601b2a2f-kube-api-access-w2lnh\") pod \"cert-manager-55b4d954d5-bjh2h\" (UID: \"e4450f6a-62e2-4f76-a3d1-4a8a601b2a2f\") " pod="cert-manager/cert-manager-55b4d954d5-bjh2h" Jul 28 22:14:05.082023 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:05.081988 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e4450f6a-62e2-4f76-a3d1-4a8a601b2a2f-bound-sa-token\") pod \"cert-manager-55b4d954d5-bjh2h\" (UID: \"e4450f6a-62e2-4f76-a3d1-4a8a601b2a2f\") " pod="cert-manager/cert-manager-55b4d954d5-bjh2h" Jul 28 22:14:05.082213 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:05.082046 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2lnh\" (UniqueName: \"kubernetes.io/projected/e4450f6a-62e2-4f76-a3d1-4a8a601b2a2f-kube-api-access-w2lnh\") pod \"cert-manager-55b4d954d5-bjh2h\" (UID: \"e4450f6a-62e2-4f76-a3d1-4a8a601b2a2f\") " pod="cert-manager/cert-manager-55b4d954d5-bjh2h" Jul 28 22:14:05.090328 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:05.090275 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e4450f6a-62e2-4f76-a3d1-4a8a601b2a2f-bound-sa-token\") pod \"cert-manager-55b4d954d5-bjh2h\" (UID: \"e4450f6a-62e2-4f76-a3d1-4a8a601b2a2f\") " pod="cert-manager/cert-manager-55b4d954d5-bjh2h" Jul 28 22:14:05.090529 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:05.090506 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2lnh\" (UniqueName: \"kubernetes.io/projected/e4450f6a-62e2-4f76-a3d1-4a8a601b2a2f-kube-api-access-w2lnh\") pod \"cert-manager-55b4d954d5-bjh2h\" (UID: \"e4450f6a-62e2-4f76-a3d1-4a8a601b2a2f\") " pod="cert-manager/cert-manager-55b4d954d5-bjh2h" Jul 28 22:14:05.256105 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:05.256076 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-bjh2h" Jul 28 22:14:05.402863 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:05.402823 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-bjh2h"] Jul 28 22:14:06.221990 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:14:06.221947 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode4450f6a_62e2_4f76_a3d1_4a8a601b2a2f.slice/crio-f857741c1591d0718579bc50bba787482e1aa3a17ba2df5a12e6a5bcfb40c186 WatchSource:0}: Error finding container f857741c1591d0718579bc50bba787482e1aa3a17ba2df5a12e6a5bcfb40c186: Status 404 returned error can't find the container with id f857741c1591d0718579bc50bba787482e1aa3a17ba2df5a12e6a5bcfb40c186 Jul 28 22:14:06.678936 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:06.678851 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-bjh2h" event={"ID":"e4450f6a-62e2-4f76-a3d1-4a8a601b2a2f","Type":"ContainerStarted","Data":"f857741c1591d0718579bc50bba787482e1aa3a17ba2df5a12e6a5bcfb40c186"} Jul 28 22:14:06.680305 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:06.680282 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-6tcc5" event={"ID":"7b0b78a4-e337-47f6-b757-d5e39eb7d7f2","Type":"ContainerStarted","Data":"fab3ccee67f3a5266b1401bfa44208e493fd1ddc168f5636a386848ff655e74d"} Jul 28 22:14:06.680412 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:06.680401 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-6tcc5" Jul 28 22:14:06.695937 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:06.695891 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-6tcc5" podStartSLOduration=1.7338650690000001 podStartE2EDuration="5.695877799s" totalImagesPullingTime="3.96201273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:14:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:14:06.694338762 +0000 UTC m=+266.781596831" watchObservedRunningTime="2026-07-28 22:14:06.695877799 +0000 UTC m=+266.783135868" Jul 28 22:14:07.686598 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:07.686559 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-bjh2h" event={"ID":"e4450f6a-62e2-4f76-a3d1-4a8a601b2a2f","Type":"ContainerStarted","Data":"337ef43e2a068a2cd8d21141ce32904c440608449ff818e3d3afb17f2a7772cb"} Jul 28 22:14:07.702069 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:07.701905 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-bjh2h" podStartSLOduration=3.165353267 podStartE2EDuration="3.70188626s" totalImagesPullingTime="536.532993ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:14:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:14:07.701597935 +0000 UTC m=+267.788856004" watchObservedRunningTime="2026-07-28 22:14:07.70188626 +0000 UTC m=+267.789144330" Jul 28 22:14:12.688894 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:12.688857 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-6tcc5" Jul 28 22:14:14.132267 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:14.132229 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affz76z"] Jul 28 22:14:14.135936 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:14.135914 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affz76z" Jul 28 22:14:14.138174 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:14.138141 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 28 22:14:14.138385 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:14.138369 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 28 22:14:14.138470 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:14.138384 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-dclqn\"" Jul 28 22:14:14.144462 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:14.144436 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affz76z"] Jul 28 22:14:14.253880 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:14.253828 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7jkn\" (UniqueName: \"kubernetes.io/projected/ee281419-8898-49bd-a053-02e1c42012dd-kube-api-access-m7jkn\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affz76z\" (UID: \"ee281419-8898-49bd-a053-02e1c42012dd\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affz76z" Jul 28 22:14:14.254069 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:14.253934 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee281419-8898-49bd-a053-02e1c42012dd-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affz76z\" (UID: \"ee281419-8898-49bd-a053-02e1c42012dd\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affz76z" Jul 28 22:14:14.254069 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:14.253957 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee281419-8898-49bd-a053-02e1c42012dd-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affz76z\" (UID: \"ee281419-8898-49bd-a053-02e1c42012dd\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affz76z" Jul 28 22:14:14.354942 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:14.354905 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7jkn\" (UniqueName: \"kubernetes.io/projected/ee281419-8898-49bd-a053-02e1c42012dd-kube-api-access-m7jkn\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affz76z\" (UID: \"ee281419-8898-49bd-a053-02e1c42012dd\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affz76z" Jul 28 22:14:14.355130 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:14.354968 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee281419-8898-49bd-a053-02e1c42012dd-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affz76z\" (UID: \"ee281419-8898-49bd-a053-02e1c42012dd\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affz76z" Jul 28 22:14:14.355130 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:14.354991 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee281419-8898-49bd-a053-02e1c42012dd-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affz76z\" (UID: \"ee281419-8898-49bd-a053-02e1c42012dd\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affz76z" Jul 28 22:14:14.355324 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:14.355307 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee281419-8898-49bd-a053-02e1c42012dd-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affz76z\" (UID: \"ee281419-8898-49bd-a053-02e1c42012dd\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affz76z" Jul 28 22:14:14.355390 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:14.355363 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee281419-8898-49bd-a053-02e1c42012dd-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affz76z\" (UID: \"ee281419-8898-49bd-a053-02e1c42012dd\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affz76z" Jul 28 22:14:14.365063 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:14.365038 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7jkn\" (UniqueName: \"kubernetes.io/projected/ee281419-8898-49bd-a053-02e1c42012dd-kube-api-access-m7jkn\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affz76z\" (UID: \"ee281419-8898-49bd-a053-02e1c42012dd\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affz76z" Jul 28 22:14:14.445914 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:14.445815 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affz76z" Jul 28 22:14:14.586403 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:14.586370 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affz76z"] Jul 28 22:14:14.589989 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:14:14.589960 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee281419_8898_49bd_a053_02e1c42012dd.slice/crio-9130b583a3b2163756f5194f201a04abc9ee15ac971f9966910e58b12af58c2c WatchSource:0}: Error finding container 9130b583a3b2163756f5194f201a04abc9ee15ac971f9966910e58b12af58c2c: Status 404 returned error can't find the container with id 9130b583a3b2163756f5194f201a04abc9ee15ac971f9966910e58b12af58c2c Jul 28 22:14:14.717831 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:14.717803 2553 generic.go:356] "Generic (PLEG): container finished" podID="ee281419-8898-49bd-a053-02e1c42012dd" containerID="3c630875acb4ce7547cf7285fabaae6403af059ab99b8db0d94eb2f3598cc0c0" exitCode=0 Jul 28 22:14:14.717990 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:14.717882 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affz76z" event={"ID":"ee281419-8898-49bd-a053-02e1c42012dd","Type":"ContainerDied","Data":"3c630875acb4ce7547cf7285fabaae6403af059ab99b8db0d94eb2f3598cc0c0"} Jul 28 22:14:14.717990 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:14.717932 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affz76z" event={"ID":"ee281419-8898-49bd-a053-02e1c42012dd","Type":"ContainerStarted","Data":"9130b583a3b2163756f5194f201a04abc9ee15ac971f9966910e58b12af58c2c"} Jul 28 22:14:15.723585 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:15.723553 2553 generic.go:356] "Generic (PLEG): container finished" podID="ee281419-8898-49bd-a053-02e1c42012dd" containerID="06304ba73238e759495cc4077b704a9175071703a2416b8c90a9a8c34baca169" exitCode=0 Jul 28 22:14:15.724073 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:15.723638 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affz76z" event={"ID":"ee281419-8898-49bd-a053-02e1c42012dd","Type":"ContainerDied","Data":"06304ba73238e759495cc4077b704a9175071703a2416b8c90a9a8c34baca169"} Jul 28 22:14:16.729507 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:16.729471 2553 generic.go:356] "Generic (PLEG): container finished" podID="ee281419-8898-49bd-a053-02e1c42012dd" containerID="631dec6388a10a11bdde3d1d1f41c2a581ecd4a04d478070c05c031dc873b193" exitCode=0 Jul 28 22:14:16.729927 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:16.729549 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affz76z" event={"ID":"ee281419-8898-49bd-a053-02e1c42012dd","Type":"ContainerDied","Data":"631dec6388a10a11bdde3d1d1f41c2a581ecd4a04d478070c05c031dc873b193"} Jul 28 22:14:17.871644 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:17.871617 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affz76z" Jul 28 22:14:17.983183 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:17.983150 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee281419-8898-49bd-a053-02e1c42012dd-bundle\") pod \"ee281419-8898-49bd-a053-02e1c42012dd\" (UID: \"ee281419-8898-49bd-a053-02e1c42012dd\") " Jul 28 22:14:17.983345 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:17.983208 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7jkn\" (UniqueName: \"kubernetes.io/projected/ee281419-8898-49bd-a053-02e1c42012dd-kube-api-access-m7jkn\") pod \"ee281419-8898-49bd-a053-02e1c42012dd\" (UID: \"ee281419-8898-49bd-a053-02e1c42012dd\") " Jul 28 22:14:17.983345 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:17.983245 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee281419-8898-49bd-a053-02e1c42012dd-util\") pod \"ee281419-8898-49bd-a053-02e1c42012dd\" (UID: \"ee281419-8898-49bd-a053-02e1c42012dd\") " Jul 28 22:14:17.983883 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:17.983855 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee281419-8898-49bd-a053-02e1c42012dd-bundle" (OuterVolumeSpecName: "bundle") pod "ee281419-8898-49bd-a053-02e1c42012dd" (UID: "ee281419-8898-49bd-a053-02e1c42012dd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:14:17.985438 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:17.985403 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee281419-8898-49bd-a053-02e1c42012dd-kube-api-access-m7jkn" (OuterVolumeSpecName: "kube-api-access-m7jkn") pod "ee281419-8898-49bd-a053-02e1c42012dd" (UID: "ee281419-8898-49bd-a053-02e1c42012dd"). InnerVolumeSpecName "kube-api-access-m7jkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:14:17.988941 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:17.988919 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee281419-8898-49bd-a053-02e1c42012dd-util" (OuterVolumeSpecName: "util") pod "ee281419-8898-49bd-a053-02e1c42012dd" (UID: "ee281419-8898-49bd-a053-02e1c42012dd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:14:18.083743 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:18.083693 2553 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee281419-8898-49bd-a053-02e1c42012dd-bundle\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:14:18.083743 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:18.083732 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7jkn\" (UniqueName: \"kubernetes.io/projected/ee281419-8898-49bd-a053-02e1c42012dd-kube-api-access-m7jkn\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:14:18.083743 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:18.083743 2553 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee281419-8898-49bd-a053-02e1c42012dd-util\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:14:18.745261 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:18.745237 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affz76z" Jul 28 22:14:18.745427 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:18.745232 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affz76z" event={"ID":"ee281419-8898-49bd-a053-02e1c42012dd","Type":"ContainerDied","Data":"9130b583a3b2163756f5194f201a04abc9ee15ac971f9966910e58b12af58c2c"} Jul 28 22:14:18.745427 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:18.745348 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9130b583a3b2163756f5194f201a04abc9ee15ac971f9966910e58b12af58c2c" Jul 28 22:14:28.043938 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:28.043903 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7b7ccc5769-88sx5"] Jul 28 22:14:28.044388 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:28.044274 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee281419-8898-49bd-a053-02e1c42012dd" containerName="extract" Jul 28 22:14:28.044388 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:28.044288 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee281419-8898-49bd-a053-02e1c42012dd" containerName="extract" Jul 28 22:14:28.044388 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:28.044296 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee281419-8898-49bd-a053-02e1c42012dd" containerName="pull" Jul 28 22:14:28.044388 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:28.044304 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee281419-8898-49bd-a053-02e1c42012dd" containerName="pull" Jul 28 22:14:28.044388 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:28.044321 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee281419-8898-49bd-a053-02e1c42012dd" containerName="util" Jul 28 22:14:28.044388 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:28.044327 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee281419-8898-49bd-a053-02e1c42012dd" containerName="util" Jul 28 22:14:28.044581 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:28.044397 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee281419-8898-49bd-a053-02e1c42012dd" containerName="extract" Jul 28 22:14:28.049972 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:28.049951 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7b7ccc5769-88sx5" Jul 28 22:14:28.054728 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:28.054679 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 28 22:14:28.054821 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:28.054807 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-zwhdk\"" Jul 28 22:14:28.055480 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:28.055464 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 28 22:14:28.055719 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:28.055690 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 28 22:14:28.058631 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:28.058480 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 28 22:14:28.073021 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:28.072994 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7b7ccc5769-88sx5"] Jul 28 22:14:28.163947 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:28.163910 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/331d730d-2853-4509-9e90-4727ad3ed40d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b7ccc5769-88sx5\" (UID: \"331d730d-2853-4509-9e90-4727ad3ed40d\") " pod="opendatahub/opendatahub-operator-controller-manager-7b7ccc5769-88sx5" Jul 28 22:14:28.164116 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:28.163974 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z77gg\" (UniqueName: \"kubernetes.io/projected/331d730d-2853-4509-9e90-4727ad3ed40d-kube-api-access-z77gg\") pod \"opendatahub-operator-controller-manager-7b7ccc5769-88sx5\" (UID: \"331d730d-2853-4509-9e90-4727ad3ed40d\") " pod="opendatahub/opendatahub-operator-controller-manager-7b7ccc5769-88sx5" Jul 28 22:14:28.164116 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:28.164025 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/331d730d-2853-4509-9e90-4727ad3ed40d-tmp\") pod \"opendatahub-operator-controller-manager-7b7ccc5769-88sx5\" (UID: \"331d730d-2853-4509-9e90-4727ad3ed40d\") " pod="opendatahub/opendatahub-operator-controller-manager-7b7ccc5769-88sx5" Jul 28 22:14:28.164116 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:28.164103 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/331d730d-2853-4509-9e90-4727ad3ed40d-manifests\") pod \"opendatahub-operator-controller-manager-7b7ccc5769-88sx5\" (UID: \"331d730d-2853-4509-9e90-4727ad3ed40d\") " pod="opendatahub/opendatahub-operator-controller-manager-7b7ccc5769-88sx5" Jul 28 22:14:28.164224 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:28.164134 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/331d730d-2853-4509-9e90-4727ad3ed40d-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b7ccc5769-88sx5\" (UID: \"331d730d-2853-4509-9e90-4727ad3ed40d\") " pod="opendatahub/opendatahub-operator-controller-manager-7b7ccc5769-88sx5" Jul 28 22:14:28.265540 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:28.265500 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z77gg\" (UniqueName: \"kubernetes.io/projected/331d730d-2853-4509-9e90-4727ad3ed40d-kube-api-access-z77gg\") pod \"opendatahub-operator-controller-manager-7b7ccc5769-88sx5\" (UID: \"331d730d-2853-4509-9e90-4727ad3ed40d\") " pod="opendatahub/opendatahub-operator-controller-manager-7b7ccc5769-88sx5" Jul 28 22:14:28.265743 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:28.265555 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/331d730d-2853-4509-9e90-4727ad3ed40d-tmp\") pod \"opendatahub-operator-controller-manager-7b7ccc5769-88sx5\" (UID: \"331d730d-2853-4509-9e90-4727ad3ed40d\") " pod="opendatahub/opendatahub-operator-controller-manager-7b7ccc5769-88sx5" Jul 28 22:14:28.265743 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:28.265599 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/331d730d-2853-4509-9e90-4727ad3ed40d-manifests\") pod \"opendatahub-operator-controller-manager-7b7ccc5769-88sx5\" (UID: \"331d730d-2853-4509-9e90-4727ad3ed40d\") " pod="opendatahub/opendatahub-operator-controller-manager-7b7ccc5769-88sx5" Jul 28 22:14:28.265743 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:28.265619 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/331d730d-2853-4509-9e90-4727ad3ed40d-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b7ccc5769-88sx5\" (UID: \"331d730d-2853-4509-9e90-4727ad3ed40d\") " pod="opendatahub/opendatahub-operator-controller-manager-7b7ccc5769-88sx5" Jul 28 22:14:28.265743 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:28.265643 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/331d730d-2853-4509-9e90-4727ad3ed40d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b7ccc5769-88sx5\" (UID: \"331d730d-2853-4509-9e90-4727ad3ed40d\") " pod="opendatahub/opendatahub-operator-controller-manager-7b7ccc5769-88sx5" Jul 28 22:14:28.266135 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:28.266111 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/331d730d-2853-4509-9e90-4727ad3ed40d-manifests\") pod \"opendatahub-operator-controller-manager-7b7ccc5769-88sx5\" (UID: \"331d730d-2853-4509-9e90-4727ad3ed40d\") " pod="opendatahub/opendatahub-operator-controller-manager-7b7ccc5769-88sx5" Jul 28 22:14:28.268020 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:28.267990 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/331d730d-2853-4509-9e90-4727ad3ed40d-tmp\") pod \"opendatahub-operator-controller-manager-7b7ccc5769-88sx5\" (UID: \"331d730d-2853-4509-9e90-4727ad3ed40d\") " pod="opendatahub/opendatahub-operator-controller-manager-7b7ccc5769-88sx5" Jul 28 22:14:28.268334 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:28.268312 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/331d730d-2853-4509-9e90-4727ad3ed40d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b7ccc5769-88sx5\" (UID: \"331d730d-2853-4509-9e90-4727ad3ed40d\") " pod="opendatahub/opendatahub-operator-controller-manager-7b7ccc5769-88sx5" Jul 28 22:14:28.268376 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:28.268329 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/331d730d-2853-4509-9e90-4727ad3ed40d-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b7ccc5769-88sx5\" (UID: \"331d730d-2853-4509-9e90-4727ad3ed40d\") " pod="opendatahub/opendatahub-operator-controller-manager-7b7ccc5769-88sx5" Jul 28 22:14:28.275591 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:28.275566 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z77gg\" (UniqueName: \"kubernetes.io/projected/331d730d-2853-4509-9e90-4727ad3ed40d-kube-api-access-z77gg\") pod \"opendatahub-operator-controller-manager-7b7ccc5769-88sx5\" (UID: \"331d730d-2853-4509-9e90-4727ad3ed40d\") " pod="opendatahub/opendatahub-operator-controller-manager-7b7ccc5769-88sx5" Jul 28 22:14:28.360466 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:28.360359 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7b7ccc5769-88sx5" Jul 28 22:14:28.519350 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:28.519324 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7b7ccc5769-88sx5"] Jul 28 22:14:28.521497 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:14:28.521453 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod331d730d_2853_4509_9e90_4727ad3ed40d.slice/crio-96d5caf2005bb261111d0bb09eceff120ca385a9d1e8a6250355e7b10c520714 WatchSource:0}: Error finding container 96d5caf2005bb261111d0bb09eceff120ca385a9d1e8a6250355e7b10c520714: Status 404 returned error can't find the container with id 96d5caf2005bb261111d0bb09eceff120ca385a9d1e8a6250355e7b10c520714 Jul 28 22:14:28.563226 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:28.563187 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-8586db4f6f-rqqn9"] Jul 28 22:14:28.568685 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:28.568664 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-8586db4f6f-rqqn9" Jul 28 22:14:28.573290 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:28.573271 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 28 22:14:28.573584 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:28.573567 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 28 22:14:28.573693 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:28.573600 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 28 22:14:28.573693 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:28.573642 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-llnh7\"" Jul 28 22:14:28.573693 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:28.573648 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 28 22:14:28.573919 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:28.573905 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 28 22:14:28.587657 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:28.587631 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-8586db4f6f-rqqn9"] Jul 28 22:14:28.669440 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:28.669347 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fae924a4-7ded-4b72-a773-beb3d4e18e84-cert\") pod \"lws-controller-manager-8586db4f6f-rqqn9\" (UID: \"fae924a4-7ded-4b72-a773-beb3d4e18e84\") " pod="openshift-lws-operator/lws-controller-manager-8586db4f6f-rqqn9" Jul 28 22:14:28.669440 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:28.669410 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fhnl\" (UniqueName: \"kubernetes.io/projected/fae924a4-7ded-4b72-a773-beb3d4e18e84-kube-api-access-6fhnl\") pod \"lws-controller-manager-8586db4f6f-rqqn9\" (UID: \"fae924a4-7ded-4b72-a773-beb3d4e18e84\") " pod="openshift-lws-operator/lws-controller-manager-8586db4f6f-rqqn9" Jul 28 22:14:28.669440 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:28.669438 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/fae924a4-7ded-4b72-a773-beb3d4e18e84-manager-config\") pod \"lws-controller-manager-8586db4f6f-rqqn9\" (UID: \"fae924a4-7ded-4b72-a773-beb3d4e18e84\") " pod="openshift-lws-operator/lws-controller-manager-8586db4f6f-rqqn9" Jul 28 22:14:28.669690 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:28.669534 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fae924a4-7ded-4b72-a773-beb3d4e18e84-metrics-cert\") pod \"lws-controller-manager-8586db4f6f-rqqn9\" (UID: \"fae924a4-7ded-4b72-a773-beb3d4e18e84\") " pod="openshift-lws-operator/lws-controller-manager-8586db4f6f-rqqn9" Jul 28 22:14:28.770912 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:28.770869 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fae924a4-7ded-4b72-a773-beb3d4e18e84-cert\") pod \"lws-controller-manager-8586db4f6f-rqqn9\" (UID: \"fae924a4-7ded-4b72-a773-beb3d4e18e84\") " pod="openshift-lws-operator/lws-controller-manager-8586db4f6f-rqqn9" Jul 28 22:14:28.771117 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:28.771018 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fhnl\" (UniqueName: \"kubernetes.io/projected/fae924a4-7ded-4b72-a773-beb3d4e18e84-kube-api-access-6fhnl\") pod \"lws-controller-manager-8586db4f6f-rqqn9\" (UID: \"fae924a4-7ded-4b72-a773-beb3d4e18e84\") " pod="openshift-lws-operator/lws-controller-manager-8586db4f6f-rqqn9" Jul 28 22:14:28.771117 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:28.771089 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/fae924a4-7ded-4b72-a773-beb3d4e18e84-manager-config\") pod \"lws-controller-manager-8586db4f6f-rqqn9\" (UID: \"fae924a4-7ded-4b72-a773-beb3d4e18e84\") " pod="openshift-lws-operator/lws-controller-manager-8586db4f6f-rqqn9" Jul 28 22:14:28.771239 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:28.771182 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fae924a4-7ded-4b72-a773-beb3d4e18e84-metrics-cert\") pod \"lws-controller-manager-8586db4f6f-rqqn9\" (UID: \"fae924a4-7ded-4b72-a773-beb3d4e18e84\") " pod="openshift-lws-operator/lws-controller-manager-8586db4f6f-rqqn9" Jul 28 22:14:28.771698 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:28.771672 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/fae924a4-7ded-4b72-a773-beb3d4e18e84-manager-config\") pod \"lws-controller-manager-8586db4f6f-rqqn9\" (UID: \"fae924a4-7ded-4b72-a773-beb3d4e18e84\") " pod="openshift-lws-operator/lws-controller-manager-8586db4f6f-rqqn9" Jul 28 22:14:28.773544 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:28.773516 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fae924a4-7ded-4b72-a773-beb3d4e18e84-cert\") pod \"lws-controller-manager-8586db4f6f-rqqn9\" (UID: \"fae924a4-7ded-4b72-a773-beb3d4e18e84\") " pod="openshift-lws-operator/lws-controller-manager-8586db4f6f-rqqn9" Jul 28 22:14:28.773624 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:28.773564 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fae924a4-7ded-4b72-a773-beb3d4e18e84-metrics-cert\") pod \"lws-controller-manager-8586db4f6f-rqqn9\" (UID: \"fae924a4-7ded-4b72-a773-beb3d4e18e84\") " pod="openshift-lws-operator/lws-controller-manager-8586db4f6f-rqqn9" Jul 28 22:14:28.780344 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:28.780324 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fhnl\" (UniqueName: \"kubernetes.io/projected/fae924a4-7ded-4b72-a773-beb3d4e18e84-kube-api-access-6fhnl\") pod \"lws-controller-manager-8586db4f6f-rqqn9\" (UID: \"fae924a4-7ded-4b72-a773-beb3d4e18e84\") " pod="openshift-lws-operator/lws-controller-manager-8586db4f6f-rqqn9" Jul 28 22:14:28.788115 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:28.788092 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b7ccc5769-88sx5" event={"ID":"331d730d-2853-4509-9e90-4727ad3ed40d","Type":"ContainerStarted","Data":"96d5caf2005bb261111d0bb09eceff120ca385a9d1e8a6250355e7b10c520714"} Jul 28 22:14:28.878543 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:28.878510 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-8586db4f6f-rqqn9" Jul 28 22:14:29.020883 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:29.020836 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-8586db4f6f-rqqn9"] Jul 28 22:14:29.023665 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:14:29.023633 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfae924a4_7ded_4b72_a773_beb3d4e18e84.slice/crio-bca5e61aa6f3a4c8c31bb6664a5eec54fc6666a04604b6947394b823bf2f141e WatchSource:0}: Error finding container bca5e61aa6f3a4c8c31bb6664a5eec54fc6666a04604b6947394b823bf2f141e: Status 404 returned error can't find the container with id bca5e61aa6f3a4c8c31bb6664a5eec54fc6666a04604b6947394b823bf2f141e Jul 28 22:14:29.796291 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:29.796248 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-8586db4f6f-rqqn9" event={"ID":"fae924a4-7ded-4b72-a773-beb3d4e18e84","Type":"ContainerStarted","Data":"bca5e61aa6f3a4c8c31bb6664a5eec54fc6666a04604b6947394b823bf2f141e"} Jul 28 22:14:32.812240 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:32.812202 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-8586db4f6f-rqqn9" event={"ID":"fae924a4-7ded-4b72-a773-beb3d4e18e84","Type":"ContainerStarted","Data":"df6e0e1ba707079d76f1e3ba61300ae1461958724ed31eb3326e69ff6fcaa346"} Jul 28 22:14:32.812779 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:32.812267 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-8586db4f6f-rqqn9" Jul 28 22:14:32.814004 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:32.813979 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b7ccc5769-88sx5" event={"ID":"331d730d-2853-4509-9e90-4727ad3ed40d","Type":"ContainerStarted","Data":"94fbc8d9c526d70a75edb485801bbebfe1ddf7c1a2c17c9ee85b1ce53f8f57d7"} Jul 28 22:14:32.830348 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:32.830296 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-8586db4f6f-rqqn9" podStartSLOduration=1.940307735 podStartE2EDuration="4.830277901s" totalImagesPullingTime="2.889970166s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:14:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:14:32.829993358 +0000 UTC m=+292.917251430" watchObservedRunningTime="2026-07-28 22:14:32.830277901 +0000 UTC m=+292.917535973" Jul 28 22:14:33.820448 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:33.820415 2553 generic.go:356] "Generic (PLEG): container finished" podID="331d730d-2853-4509-9e90-4727ad3ed40d" containerID="94fbc8d9c526d70a75edb485801bbebfe1ddf7c1a2c17c9ee85b1ce53f8f57d7" exitCode=0 Jul 28 22:14:33.820855 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:33.820505 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b7ccc5769-88sx5" event={"ID":"331d730d-2853-4509-9e90-4727ad3ed40d","Type":"ContainerDied","Data":"94fbc8d9c526d70a75edb485801bbebfe1ddf7c1a2c17c9ee85b1ce53f8f57d7"} Jul 28 22:14:34.826946 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:34.826910 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b7ccc5769-88sx5" event={"ID":"331d730d-2853-4509-9e90-4727ad3ed40d","Type":"ContainerStarted","Data":"d327c99a33840d357948ca73c33a37b5afa641cb25807881ac5a894115dc89e4"} Jul 28 22:14:34.827363 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:34.827015 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7b7ccc5769-88sx5" Jul 28 22:14:34.849236 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:34.849175 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7b7ccc5769-88sx5" podStartSLOduration=1.9579156530000001 podStartE2EDuration="6.849159206s" totalImagesPullingTime="3.776505069s" totalInitContainerRuntime="1.114738484s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:14:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:14:34.847178331 +0000 UTC m=+294.934436401" watchObservedRunningTime="2026-07-28 22:14:34.849159206 +0000 UTC m=+294.936417274" Jul 28 22:14:40.274782 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:40.274749 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mldhp_e1ced996-f136-4d3e-9c85-a5b7809b4517/console-operator/1.log" Jul 28 22:14:40.278013 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:40.277983 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mldhp_e1ced996-f136-4d3e-9c85-a5b7809b4517/console-operator/1.log" Jul 28 22:14:40.285179 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:40.285157 2553 kubelet.go:1648] "Image garbage collection succeeded" Jul 28 22:14:43.822941 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:43.822907 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-8586db4f6f-rqqn9" Jul 28 22:14:45.833760 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:45.833733 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7b7ccc5769-88sx5" Jul 28 22:14:56.346816 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:56.346777 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x"] Jul 28 22:14:56.351470 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:56.351445 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x" Jul 28 22:14:56.353878 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:56.353856 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 28 22:14:56.354020 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:56.353970 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-7n8wj\"" Jul 28 22:14:56.364514 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:56.364487 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x"] Jul 28 22:14:56.500221 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:56.500190 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74a8aa18-d749-4151-a539-177b8fb320a3-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x\" (UID: \"74a8aa18-d749-4151-a539-177b8fb320a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x" Jul 28 22:14:56.500221 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:56.500228 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qzjk\" (UniqueName: \"kubernetes.io/projected/74a8aa18-d749-4151-a539-177b8fb320a3-kube-api-access-9qzjk\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x\" (UID: \"74a8aa18-d749-4151-a539-177b8fb320a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x" Jul 28 22:14:56.500442 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:56.500257 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74a8aa18-d749-4151-a539-177b8fb320a3-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x\" (UID: \"74a8aa18-d749-4151-a539-177b8fb320a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x" Jul 28 22:14:56.500442 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:56.500335 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74a8aa18-d749-4151-a539-177b8fb320a3-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x\" (UID: \"74a8aa18-d749-4151-a539-177b8fb320a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x" Jul 28 22:14:56.500442 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:56.500372 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74a8aa18-d749-4151-a539-177b8fb320a3-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x\" (UID: \"74a8aa18-d749-4151-a539-177b8fb320a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x" Jul 28 22:14:56.500442 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:56.500436 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a8aa18-d749-4151-a539-177b8fb320a3-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x\" (UID: \"74a8aa18-d749-4151-a539-177b8fb320a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x" Jul 28 22:14:56.500566 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:56.500452 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74a8aa18-d749-4151-a539-177b8fb320a3-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x\" (UID: \"74a8aa18-d749-4151-a539-177b8fb320a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x" Jul 28 22:14:56.500566 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:56.500486 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a8aa18-d749-4151-a539-177b8fb320a3-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x\" (UID: \"74a8aa18-d749-4151-a539-177b8fb320a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x" Jul 28 22:14:56.500566 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:56.500517 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74a8aa18-d749-4151-a539-177b8fb320a3-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x\" (UID: \"74a8aa18-d749-4151-a539-177b8fb320a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x" Jul 28 22:14:56.601351 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:56.601238 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qzjk\" (UniqueName: \"kubernetes.io/projected/74a8aa18-d749-4151-a539-177b8fb320a3-kube-api-access-9qzjk\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x\" (UID: \"74a8aa18-d749-4151-a539-177b8fb320a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x" Jul 28 22:14:56.601351 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:56.601295 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74a8aa18-d749-4151-a539-177b8fb320a3-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x\" (UID: \"74a8aa18-d749-4151-a539-177b8fb320a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x" Jul 28 22:14:56.601351 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:56.601325 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74a8aa18-d749-4151-a539-177b8fb320a3-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x\" (UID: \"74a8aa18-d749-4151-a539-177b8fb320a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x" Jul 28 22:14:56.601351 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:56.601341 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74a8aa18-d749-4151-a539-177b8fb320a3-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x\" (UID: \"74a8aa18-d749-4151-a539-177b8fb320a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x" Jul 28 22:14:56.601693 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:56.601372 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a8aa18-d749-4151-a539-177b8fb320a3-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x\" (UID: \"74a8aa18-d749-4151-a539-177b8fb320a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x" Jul 28 22:14:56.601693 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:56.601400 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74a8aa18-d749-4151-a539-177b8fb320a3-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x\" (UID: \"74a8aa18-d749-4151-a539-177b8fb320a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x" Jul 28 22:14:56.601693 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:56.601431 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a8aa18-d749-4151-a539-177b8fb320a3-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x\" (UID: \"74a8aa18-d749-4151-a539-177b8fb320a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x" Jul 28 22:14:56.601693 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:56.601476 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74a8aa18-d749-4151-a539-177b8fb320a3-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x\" (UID: \"74a8aa18-d749-4151-a539-177b8fb320a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x" Jul 28 22:14:56.601693 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:56.601505 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74a8aa18-d749-4151-a539-177b8fb320a3-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x\" (UID: \"74a8aa18-d749-4151-a539-177b8fb320a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x" Jul 28 22:14:56.602002 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:56.601839 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a8aa18-d749-4151-a539-177b8fb320a3-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x\" (UID: \"74a8aa18-d749-4151-a539-177b8fb320a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x" Jul 28 22:14:56.602061 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:56.602007 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74a8aa18-d749-4151-a539-177b8fb320a3-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x\" (UID: \"74a8aa18-d749-4151-a539-177b8fb320a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x" Jul 28 22:14:56.602122 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:56.602072 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74a8aa18-d749-4151-a539-177b8fb320a3-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x\" (UID: \"74a8aa18-d749-4151-a539-177b8fb320a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x" Jul 28 22:14:56.602162 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:56.602116 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a8aa18-d749-4151-a539-177b8fb320a3-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x\" (UID: \"74a8aa18-d749-4151-a539-177b8fb320a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x" Jul 28 22:14:56.602409 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:56.602387 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74a8aa18-d749-4151-a539-177b8fb320a3-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x\" (UID: \"74a8aa18-d749-4151-a539-177b8fb320a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x" Jul 28 22:14:56.603892 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:56.603870 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74a8aa18-d749-4151-a539-177b8fb320a3-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x\" (UID: \"74a8aa18-d749-4151-a539-177b8fb320a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x" Jul 28 22:14:56.604115 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:56.604093 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74a8aa18-d749-4151-a539-177b8fb320a3-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x\" (UID: \"74a8aa18-d749-4151-a539-177b8fb320a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x" Jul 28 22:14:56.609691 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:56.609668 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74a8aa18-d749-4151-a539-177b8fb320a3-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x\" (UID: \"74a8aa18-d749-4151-a539-177b8fb320a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x" Jul 28 22:14:56.609946 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:56.609925 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qzjk\" (UniqueName: \"kubernetes.io/projected/74a8aa18-d749-4151-a539-177b8fb320a3-kube-api-access-9qzjk\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x\" (UID: \"74a8aa18-d749-4151-a539-177b8fb320a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x" Jul 28 22:14:56.663407 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:56.663374 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x" Jul 28 22:14:56.801694 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:56.801666 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x"] Jul 28 22:14:56.803696 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:14:56.803667 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74a8aa18_d749_4151_a539_177b8fb320a3.slice/crio-b855749efadb71226bd3516561de3c22c902902a42154d766b34c3141031f3b1 WatchSource:0}: Error finding container b855749efadb71226bd3516561de3c22c902902a42154d766b34c3141031f3b1: Status 404 returned error can't find the container with id b855749efadb71226bd3516561de3c22c902902a42154d766b34c3141031f3b1 Jul 28 22:14:56.805776 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:56.805760 2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 22:14:56.926994 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:56.926919 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x" event={"ID":"74a8aa18-d749-4151-a539-177b8fb320a3","Type":"ContainerStarted","Data":"b855749efadb71226bd3516561de3c22c902902a42154d766b34c3141031f3b1"} Jul 28 22:14:59.010197 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:59.010159 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-vdkjj"] Jul 28 22:14:59.017578 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:59.017548 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vdkjj" Jul 28 22:14:59.019786 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:59.019758 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 28 22:14:59.019786 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:59.019761 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 28 22:14:59.019970 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:59.019792 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-g2wcz\"" Jul 28 22:14:59.019970 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:59.019868 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 28 22:14:59.024985 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:59.024951 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-vdkjj"] Jul 28 22:14:59.123257 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:59.123207 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c283e4ed-82e2-474a-b100-c34afa239b17-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-vdkjj\" (UID: \"c283e4ed-82e2-474a-b100-c34afa239b17\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vdkjj" Jul 28 22:14:59.123446 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:59.123315 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c283e4ed-82e2-474a-b100-c34afa239b17-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-vdkjj\" (UID: \"c283e4ed-82e2-474a-b100-c34afa239b17\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vdkjj" Jul 28 22:14:59.123446 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:59.123403 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkwfv\" (UniqueName: \"kubernetes.io/projected/c283e4ed-82e2-474a-b100-c34afa239b17-kube-api-access-vkwfv\") pod \"istiod-openshift-gateway-5677ccb49b-vdkjj\" (UID: \"c283e4ed-82e2-474a-b100-c34afa239b17\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vdkjj" Jul 28 22:14:59.123576 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:59.123494 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c283e4ed-82e2-474a-b100-c34afa239b17-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-vdkjj\" (UID: \"c283e4ed-82e2-474a-b100-c34afa239b17\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vdkjj" Jul 28 22:14:59.123576 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:59.123533 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c283e4ed-82e2-474a-b100-c34afa239b17-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-vdkjj\" (UID: \"c283e4ed-82e2-474a-b100-c34afa239b17\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vdkjj" Jul 28 22:14:59.123576 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:59.123557 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c283e4ed-82e2-474a-b100-c34afa239b17-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-vdkjj\" (UID: \"c283e4ed-82e2-474a-b100-c34afa239b17\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vdkjj" Jul 28 22:14:59.123739 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:59.123629 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c283e4ed-82e2-474a-b100-c34afa239b17-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-vdkjj\" (UID: \"c283e4ed-82e2-474a-b100-c34afa239b17\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vdkjj" Jul 28 22:14:59.224620 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:59.224587 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c283e4ed-82e2-474a-b100-c34afa239b17-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-vdkjj\" (UID: \"c283e4ed-82e2-474a-b100-c34afa239b17\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vdkjj" Jul 28 22:14:59.224818 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:59.224662 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkwfv\" (UniqueName: \"kubernetes.io/projected/c283e4ed-82e2-474a-b100-c34afa239b17-kube-api-access-vkwfv\") pod \"istiod-openshift-gateway-5677ccb49b-vdkjj\" (UID: \"c283e4ed-82e2-474a-b100-c34afa239b17\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vdkjj" Jul 28 22:14:59.224818 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:59.224744 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c283e4ed-82e2-474a-b100-c34afa239b17-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-vdkjj\" (UID: \"c283e4ed-82e2-474a-b100-c34afa239b17\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vdkjj" Jul 28 22:14:59.224818 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:59.224790 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c283e4ed-82e2-474a-b100-c34afa239b17-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-vdkjj\" (UID: \"c283e4ed-82e2-474a-b100-c34afa239b17\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vdkjj" Jul 28 22:14:59.224818 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:59.224808 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c283e4ed-82e2-474a-b100-c34afa239b17-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-vdkjj\" (UID: \"c283e4ed-82e2-474a-b100-c34afa239b17\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vdkjj" Jul 28 22:14:59.225028 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:59.224840 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c283e4ed-82e2-474a-b100-c34afa239b17-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-vdkjj\" (UID: \"c283e4ed-82e2-474a-b100-c34afa239b17\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vdkjj" Jul 28 22:14:59.225028 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:59.224890 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c283e4ed-82e2-474a-b100-c34afa239b17-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-vdkjj\" (UID: \"c283e4ed-82e2-474a-b100-c34afa239b17\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vdkjj" Jul 28 22:14:59.225801 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:59.225674 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c283e4ed-82e2-474a-b100-c34afa239b17-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-vdkjj\" (UID: \"c283e4ed-82e2-474a-b100-c34afa239b17\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vdkjj" Jul 28 22:14:59.227681 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:59.227655 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c283e4ed-82e2-474a-b100-c34afa239b17-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-vdkjj\" (UID: \"c283e4ed-82e2-474a-b100-c34afa239b17\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vdkjj" Jul 28 22:14:59.228003 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:59.227982 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c283e4ed-82e2-474a-b100-c34afa239b17-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-vdkjj\" (UID: \"c283e4ed-82e2-474a-b100-c34afa239b17\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vdkjj" Jul 28 22:14:59.228234 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:59.228214 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c283e4ed-82e2-474a-b100-c34afa239b17-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-vdkjj\" (UID: \"c283e4ed-82e2-474a-b100-c34afa239b17\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vdkjj" Jul 28 22:14:59.228305 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:59.228284 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c283e4ed-82e2-474a-b100-c34afa239b17-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-vdkjj\" (UID: \"c283e4ed-82e2-474a-b100-c34afa239b17\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vdkjj" Jul 28 22:14:59.232762 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:59.232743 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c283e4ed-82e2-474a-b100-c34afa239b17-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-vdkjj\" (UID: \"c283e4ed-82e2-474a-b100-c34afa239b17\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vdkjj" Jul 28 22:14:59.233575 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:59.233550 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkwfv\" (UniqueName: \"kubernetes.io/projected/c283e4ed-82e2-474a-b100-c34afa239b17-kube-api-access-vkwfv\") pod \"istiod-openshift-gateway-5677ccb49b-vdkjj\" (UID: \"c283e4ed-82e2-474a-b100-c34afa239b17\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vdkjj" Jul 28 22:14:59.330408 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:59.330326 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vdkjj" Jul 28 22:14:59.667468 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:59.667426 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:14:59.667598 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:59.667509 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:14:59.667598 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:59.667538 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:14:59.749682 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:59.749649 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-vdkjj"] Jul 28 22:14:59.752464 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:14:59.752441 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc283e4ed_82e2_474a_b100_c34afa239b17.slice/crio-0b261ef512d55e0cfd391a247d8094f2f5079faf6d46cf724b8704455dcd169c WatchSource:0}: Error finding container 0b261ef512d55e0cfd391a247d8094f2f5079faf6d46cf724b8704455dcd169c: Status 404 returned error can't find the container with id 0b261ef512d55e0cfd391a247d8094f2f5079faf6d46cf724b8704455dcd169c Jul 28 22:14:59.944314 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:59.944219 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x" event={"ID":"74a8aa18-d749-4151-a539-177b8fb320a3","Type":"ContainerStarted","Data":"afd1acbe7c2fa2d128a6ccd54aefc38e819ca4d030ca43b2f4d8afd22fce19e4"} Jul 28 22:14:59.945515 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:59.945491 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vdkjj" event={"ID":"c283e4ed-82e2-474a-b100-c34afa239b17","Type":"ContainerStarted","Data":"0b261ef512d55e0cfd391a247d8094f2f5079faf6d46cf724b8704455dcd169c"} Jul 28 22:14:59.966309 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:14:59.966265 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x" podStartSLOduration=1.10504507 podStartE2EDuration="3.966251043s" totalImagesPullingTime="2.861205973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:14:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:14:59.964868475 +0000 UTC m=+320.052126570" watchObservedRunningTime="2026-07-28 22:14:59.966251043 +0000 UTC m=+320.053509112" Jul 28 22:15:00.663741 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:15:00.663688 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x" Jul 28 22:15:00.665402 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:15:00.665370 2553 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Jul 28 22:15:00.665543 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:15:00.665426 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x" podUID="74a8aa18-d749-4151-a539-177b8fb320a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Jul 28 22:15:01.664205 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:15:01.664169 2553 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Jul 28 22:15:01.664683 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:15:01.664248 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x" podUID="74a8aa18-d749-4151-a539-177b8fb320a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Jul 28 22:15:01.841684 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:15:01.841645 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x"] Jul 28 22:15:02.324224 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:15:02.324179 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:15:02.324318 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:15:02.324279 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:15:02.664270 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:15:02.664189 2553 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Jul 28 22:15:02.664270 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:15:02.664244 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x" podUID="74a8aa18-d749-4151-a539-177b8fb320a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Jul 28 22:15:02.965093 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:15:02.965053 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vdkjj" event={"ID":"c283e4ed-82e2-474a-b100-c34afa239b17","Type":"ContainerStarted","Data":"3b4245be78f8e3c047118a774293872fb6572c53adaaef5c367943bd35ac764f"} Jul 28 22:15:02.965291 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:15:02.965148 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vdkjj" Jul 28 22:15:02.965291 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:15:02.965233 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x" podUID="74a8aa18-d749-4151-a539-177b8fb320a3" containerName="istio-proxy" containerID="cri-o://afd1acbe7c2fa2d128a6ccd54aefc38e819ca4d030ca43b2f4d8afd22fce19e4" gracePeriod=30 Jul 28 22:15:03.009066 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:15:03.009007 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vdkjj" podStartSLOduration=2.439394497 podStartE2EDuration="5.008988588s" totalImagesPullingTime="2.569594091s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:14:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:15:03.006145823 +0000 UTC m=+323.093403908" watchObservedRunningTime="2026-07-28 22:15:03.008988588 +0000 UTC m=+323.096246657" Jul 28 22:15:03.972305 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:15:03.972279 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vdkjj" Jul 28 22:15:08.218799 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:15:08.218776 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x" Jul 28 22:15:08.306377 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:15:08.306289 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74a8aa18-d749-4151-a539-177b8fb320a3-istio-token\") pod \"74a8aa18-d749-4151-a539-177b8fb320a3\" (UID: \"74a8aa18-d749-4151-a539-177b8fb320a3\") " Jul 28 22:15:08.306377 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:15:08.306338 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a8aa18-d749-4151-a539-177b8fb320a3-credential-socket\") pod \"74a8aa18-d749-4151-a539-177b8fb320a3\" (UID: \"74a8aa18-d749-4151-a539-177b8fb320a3\") " Jul 28 22:15:08.306377 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:15:08.306375 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a8aa18-d749-4151-a539-177b8fb320a3-workload-socket\") pod \"74a8aa18-d749-4151-a539-177b8fb320a3\" (UID: \"74a8aa18-d749-4151-a539-177b8fb320a3\") " Jul 28 22:15:08.306726 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:15:08.306405 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74a8aa18-d749-4151-a539-177b8fb320a3-istiod-ca-cert\") pod \"74a8aa18-d749-4151-a539-177b8fb320a3\" (UID: \"74a8aa18-d749-4151-a539-177b8fb320a3\") " Jul 28 22:15:08.306726 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:15:08.306449 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74a8aa18-d749-4151-a539-177b8fb320a3-istio-data\") pod \"74a8aa18-d749-4151-a539-177b8fb320a3\" (UID: \"74a8aa18-d749-4151-a539-177b8fb320a3\") " Jul 28 22:15:08.306726 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:15:08.306532 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9qzjk\" (UniqueName: \"kubernetes.io/projected/74a8aa18-d749-4151-a539-177b8fb320a3-kube-api-access-9qzjk\") pod \"74a8aa18-d749-4151-a539-177b8fb320a3\" (UID: \"74a8aa18-d749-4151-a539-177b8fb320a3\") " Jul 28 22:15:08.306726 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:15:08.306623 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74a8aa18-d749-4151-a539-177b8fb320a3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "74a8aa18-d749-4151-a539-177b8fb320a3" (UID: "74a8aa18-d749-4151-a539-177b8fb320a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:15:08.306726 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:15:08.306637 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74a8aa18-d749-4151-a539-177b8fb320a3-istio-podinfo\") pod \"74a8aa18-d749-4151-a539-177b8fb320a3\" (UID: \"74a8aa18-d749-4151-a539-177b8fb320a3\") " Jul 28 22:15:08.306726 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:15:08.306676 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74a8aa18-d749-4151-a539-177b8fb320a3-istio-envoy\") pod \"74a8aa18-d749-4151-a539-177b8fb320a3\" (UID: \"74a8aa18-d749-4151-a539-177b8fb320a3\") " Jul 28 22:15:08.306726 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:15:08.306675 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74a8aa18-d749-4151-a539-177b8fb320a3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "74a8aa18-d749-4151-a539-177b8fb320a3" (UID: "74a8aa18-d749-4151-a539-177b8fb320a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:15:08.307089 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:15:08.306723 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74a8aa18-d749-4151-a539-177b8fb320a3-workload-certs\") pod \"74a8aa18-d749-4151-a539-177b8fb320a3\" (UID: \"74a8aa18-d749-4151-a539-177b8fb320a3\") " Jul 28 22:15:08.307089 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:15:08.306756 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74a8aa18-d749-4151-a539-177b8fb320a3-istio-data" (OuterVolumeSpecName: "istio-data") pod "74a8aa18-d749-4151-a539-177b8fb320a3" (UID: "74a8aa18-d749-4151-a539-177b8fb320a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:15:08.307089 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:15:08.306835 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74a8aa18-d749-4151-a539-177b8fb320a3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "74a8aa18-d749-4151-a539-177b8fb320a3" (UID: "74a8aa18-d749-4151-a539-177b8fb320a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 22:15:08.307089 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:15:08.306958 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74a8aa18-d749-4151-a539-177b8fb320a3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "74a8aa18-d749-4151-a539-177b8fb320a3" (UID: "74a8aa18-d749-4151-a539-177b8fb320a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:15:08.307089 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:15:08.307076 2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74a8aa18-d749-4151-a539-177b8fb320a3-workload-certs\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:15:08.307296 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:15:08.307099 2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a8aa18-d749-4151-a539-177b8fb320a3-credential-socket\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:15:08.307296 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:15:08.307119 2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a8aa18-d749-4151-a539-177b8fb320a3-workload-socket\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:15:08.307296 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:15:08.307134 2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74a8aa18-d749-4151-a539-177b8fb320a3-istiod-ca-cert\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:15:08.307296 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:15:08.307146 2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74a8aa18-d749-4151-a539-177b8fb320a3-istio-data\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:15:08.308780 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:15:08.308758 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74a8aa18-d749-4151-a539-177b8fb320a3-istio-token" (OuterVolumeSpecName: "istio-token") pod "74a8aa18-d749-4151-a539-177b8fb320a3" (UID: "74a8aa18-d749-4151-a539-177b8fb320a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:15:08.308926 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:15:08.308908 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74a8aa18-d749-4151-a539-177b8fb320a3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "74a8aa18-d749-4151-a539-177b8fb320a3" (UID: "74a8aa18-d749-4151-a539-177b8fb320a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:15:08.308975 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:15:08.308959 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74a8aa18-d749-4151-a539-177b8fb320a3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "74a8aa18-d749-4151-a539-177b8fb320a3" (UID: "74a8aa18-d749-4151-a539-177b8fb320a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 22:15:08.309114 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:15:08.309098 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74a8aa18-d749-4151-a539-177b8fb320a3-kube-api-access-9qzjk" (OuterVolumeSpecName: "kube-api-access-9qzjk") pod "74a8aa18-d749-4151-a539-177b8fb320a3" (UID: "74a8aa18-d749-4151-a539-177b8fb320a3"). InnerVolumeSpecName "kube-api-access-9qzjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:15:08.407814 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:15:08.407786 2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74a8aa18-d749-4151-a539-177b8fb320a3-istio-envoy\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:15:08.407814 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:15:08.407808 2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74a8aa18-d749-4151-a539-177b8fb320a3-istio-token\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:15:08.407814 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:15:08.407818 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qzjk\" (UniqueName: \"kubernetes.io/projected/74a8aa18-d749-4151-a539-177b8fb320a3-kube-api-access-9qzjk\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:15:08.408031 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:15:08.407833 2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74a8aa18-d749-4151-a539-177b8fb320a3-istio-podinfo\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:15:08.993716 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:15:08.993665 2553 generic.go:356] "Generic (PLEG): container finished" podID="74a8aa18-d749-4151-a539-177b8fb320a3" containerID="afd1acbe7c2fa2d128a6ccd54aefc38e819ca4d030ca43b2f4d8afd22fce19e4" exitCode=0 Jul 28 22:15:08.993914 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:15:08.993737 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x" Jul 28 22:15:08.993914 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:15:08.993758 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x" event={"ID":"74a8aa18-d749-4151-a539-177b8fb320a3","Type":"ContainerDied","Data":"afd1acbe7c2fa2d128a6ccd54aefc38e819ca4d030ca43b2f4d8afd22fce19e4"} Jul 28 22:15:08.993914 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:15:08.993810 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x" event={"ID":"74a8aa18-d749-4151-a539-177b8fb320a3","Type":"ContainerDied","Data":"b855749efadb71226bd3516561de3c22c902902a42154d766b34c3141031f3b1"} Jul 28 22:15:08.993914 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:15:08.993834 2553 scope.go:117] "RemoveContainer" containerID="afd1acbe7c2fa2d128a6ccd54aefc38e819ca4d030ca43b2f4d8afd22fce19e4" Jul 28 22:15:09.002547 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:15:09.002532 2553 scope.go:117] "RemoveContainer" containerID="afd1acbe7c2fa2d128a6ccd54aefc38e819ca4d030ca43b2f4d8afd22fce19e4" Jul 28 22:15:09.002887 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:15:09.002867 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afd1acbe7c2fa2d128a6ccd54aefc38e819ca4d030ca43b2f4d8afd22fce19e4\": container with ID starting with afd1acbe7c2fa2d128a6ccd54aefc38e819ca4d030ca43b2f4d8afd22fce19e4 not found: ID does not exist" containerID="afd1acbe7c2fa2d128a6ccd54aefc38e819ca4d030ca43b2f4d8afd22fce19e4" Jul 28 22:15:09.002949 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:15:09.002894 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afd1acbe7c2fa2d128a6ccd54aefc38e819ca4d030ca43b2f4d8afd22fce19e4"} err="failed to get container status \"afd1acbe7c2fa2d128a6ccd54aefc38e819ca4d030ca43b2f4d8afd22fce19e4\": rpc error: code = NotFound desc = could not find container \"afd1acbe7c2fa2d128a6ccd54aefc38e819ca4d030ca43b2f4d8afd22fce19e4\": container with ID starting with afd1acbe7c2fa2d128a6ccd54aefc38e819ca4d030ca43b2f4d8afd22fce19e4 not found: ID does not exist" Jul 28 22:15:09.012292 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:15:09.012270 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x"] Jul 28 22:15:09.016642 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:15:09.016621 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc5cb6x"] Jul 28 22:15:10.404322 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:15:10.404286 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74a8aa18-d749-4151-a539-177b8fb320a3" path="/var/lib/kubelet/pods/74a8aa18-d749-4151-a539-177b8fb320a3/volumes" Jul 28 22:16:05.307077 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:05.307042 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1frrnl"] Jul 28 22:16:05.307531 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:05.307363 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74a8aa18-d749-4151-a539-177b8fb320a3" containerName="istio-proxy" Jul 28 22:16:05.307531 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:05.307373 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="74a8aa18-d749-4151-a539-177b8fb320a3" containerName="istio-proxy" Jul 28 22:16:05.307531 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:05.307424 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="74a8aa18-d749-4151-a539-177b8fb320a3" containerName="istio-proxy" Jul 28 22:16:05.310407 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:05.310391 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1frrnl" Jul 28 22:16:05.312596 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:05.312575 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 28 22:16:05.312744 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:05.312632 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 28 22:16:05.313093 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:05.313078 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-j2rx6\"" Jul 28 22:16:05.320227 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:05.320195 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1frrnl"] Jul 28 22:16:05.371865 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:05.371835 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/39ea32e0-a0d7-434f-832d-3a934734b409-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1frrnl\" (UID: \"39ea32e0-a0d7-434f-832d-3a934734b409\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1frrnl" Jul 28 22:16:05.371996 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:05.371921 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5csfg\" (UniqueName: \"kubernetes.io/projected/39ea32e0-a0d7-434f-832d-3a934734b409-kube-api-access-5csfg\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1frrnl\" (UID: \"39ea32e0-a0d7-434f-832d-3a934734b409\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1frrnl" Jul 28 22:16:05.371996 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:05.371988 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/39ea32e0-a0d7-434f-832d-3a934734b409-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1frrnl\" (UID: \"39ea32e0-a0d7-434f-832d-3a934734b409\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1frrnl" Jul 28 22:16:05.472490 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:05.472455 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/39ea32e0-a0d7-434f-832d-3a934734b409-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1frrnl\" (UID: \"39ea32e0-a0d7-434f-832d-3a934734b409\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1frrnl" Jul 28 22:16:05.472649 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:05.472503 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5csfg\" (UniqueName: \"kubernetes.io/projected/39ea32e0-a0d7-434f-832d-3a934734b409-kube-api-access-5csfg\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1frrnl\" (UID: \"39ea32e0-a0d7-434f-832d-3a934734b409\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1frrnl" Jul 28 22:16:05.472649 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:05.472546 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/39ea32e0-a0d7-434f-832d-3a934734b409-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1frrnl\" (UID: \"39ea32e0-a0d7-434f-832d-3a934734b409\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1frrnl" Jul 28 22:16:05.472909 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:05.472893 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/39ea32e0-a0d7-434f-832d-3a934734b409-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1frrnl\" (UID: \"39ea32e0-a0d7-434f-832d-3a934734b409\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1frrnl" Jul 28 22:16:05.472953 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:05.472892 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/39ea32e0-a0d7-434f-832d-3a934734b409-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1frrnl\" (UID: \"39ea32e0-a0d7-434f-832d-3a934734b409\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1frrnl" Jul 28 22:16:05.479998 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:05.479978 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5csfg\" (UniqueName: \"kubernetes.io/projected/39ea32e0-a0d7-434f-832d-3a934734b409-kube-api-access-5csfg\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1frrnl\" (UID: \"39ea32e0-a0d7-434f-832d-3a934734b409\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1frrnl" Jul 28 22:16:05.622015 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:05.621932 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1frrnl" Jul 28 22:16:05.754248 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:05.754218 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1frrnl"] Jul 28 22:16:05.756436 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:16:05.756410 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39ea32e0_a0d7_434f_832d_3a934734b409.slice/crio-39407de9d7fcfea18b6d3c0ceed5287a5d5875247d0de0baebe0fe59335ba265 WatchSource:0}: Error finding container 39407de9d7fcfea18b6d3c0ceed5287a5d5875247d0de0baebe0fe59335ba265: Status 404 returned error can't find the container with id 39407de9d7fcfea18b6d3c0ceed5287a5d5875247d0de0baebe0fe59335ba265 Jul 28 22:16:06.102436 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:06.102405 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p4rp6"] Jul 28 22:16:06.105877 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:06.105861 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p4rp6" Jul 28 22:16:06.112880 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:06.112862 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p4rp6"] Jul 28 22:16:06.178165 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:06.178124 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpfst\" (UniqueName: \"kubernetes.io/projected/9ecff602-ff0f-4275-92e1-24ed7c903af2-kube-api-access-kpfst\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p4rp6\" (UID: \"9ecff602-ff0f-4275-92e1-24ed7c903af2\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p4rp6" Jul 28 22:16:06.178334 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:06.178188 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9ecff602-ff0f-4275-92e1-24ed7c903af2-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p4rp6\" (UID: \"9ecff602-ff0f-4275-92e1-24ed7c903af2\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p4rp6" Jul 28 22:16:06.178334 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:06.178222 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9ecff602-ff0f-4275-92e1-24ed7c903af2-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p4rp6\" (UID: \"9ecff602-ff0f-4275-92e1-24ed7c903af2\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p4rp6" Jul 28 22:16:06.261551 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:06.261525 2553 generic.go:356] "Generic (PLEG): container finished" podID="39ea32e0-a0d7-434f-832d-3a934734b409" containerID="0238aba0715230e0409ba4d80538e527d7d90808e47f2a087141276f19e6aaa9" exitCode=0 Jul 28 22:16:06.261732 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:06.261605 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1frrnl" event={"ID":"39ea32e0-a0d7-434f-832d-3a934734b409","Type":"ContainerDied","Data":"0238aba0715230e0409ba4d80538e527d7d90808e47f2a087141276f19e6aaa9"} Jul 28 22:16:06.261732 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:06.261649 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1frrnl" event={"ID":"39ea32e0-a0d7-434f-832d-3a934734b409","Type":"ContainerStarted","Data":"39407de9d7fcfea18b6d3c0ceed5287a5d5875247d0de0baebe0fe59335ba265"} Jul 28 22:16:06.279265 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:06.279223 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9ecff602-ff0f-4275-92e1-24ed7c903af2-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p4rp6\" (UID: \"9ecff602-ff0f-4275-92e1-24ed7c903af2\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p4rp6" Jul 28 22:16:06.279389 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:06.279293 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpfst\" (UniqueName: \"kubernetes.io/projected/9ecff602-ff0f-4275-92e1-24ed7c903af2-kube-api-access-kpfst\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p4rp6\" (UID: \"9ecff602-ff0f-4275-92e1-24ed7c903af2\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p4rp6" Jul 28 22:16:06.279389 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:06.279356 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9ecff602-ff0f-4275-92e1-24ed7c903af2-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p4rp6\" (UID: \"9ecff602-ff0f-4275-92e1-24ed7c903af2\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p4rp6" Jul 28 22:16:06.279566 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:06.279548 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9ecff602-ff0f-4275-92e1-24ed7c903af2-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p4rp6\" (UID: \"9ecff602-ff0f-4275-92e1-24ed7c903af2\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p4rp6" Jul 28 22:16:06.279665 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:06.279647 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9ecff602-ff0f-4275-92e1-24ed7c903af2-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p4rp6\" (UID: \"9ecff602-ff0f-4275-92e1-24ed7c903af2\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p4rp6" Jul 28 22:16:06.286853 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:06.286834 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpfst\" (UniqueName: \"kubernetes.io/projected/9ecff602-ff0f-4275-92e1-24ed7c903af2-kube-api-access-kpfst\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p4rp6\" (UID: \"9ecff602-ff0f-4275-92e1-24ed7c903af2\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p4rp6" Jul 28 22:16:06.416368 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:06.416299 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p4rp6" Jul 28 22:16:06.548286 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:06.548259 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p4rp6"] Jul 28 22:16:06.549969 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:16:06.549938 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ecff602_ff0f_4275_92e1_24ed7c903af2.slice/crio-a096484080ee219d5e18cc493b03c4a2510cef5d12b11286d1aef60979ce83cb WatchSource:0}: Error finding container a096484080ee219d5e18cc493b03c4a2510cef5d12b11286d1aef60979ce83cb: Status 404 returned error can't find the container with id a096484080ee219d5e18cc493b03c4a2510cef5d12b11286d1aef60979ce83cb Jul 28 22:16:06.706312 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:06.706241 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hm82z"] Jul 28 22:16:06.709677 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:06.709661 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hm82z" Jul 28 22:16:06.717898 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:06.717875 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hm82z"] Jul 28 22:16:06.783293 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:06.783256 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a2c4df61-0316-4955-97dc-84b857a09e45-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hm82z\" (UID: \"a2c4df61-0316-4955-97dc-84b857a09e45\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hm82z" Jul 28 22:16:06.783454 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:06.783377 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7nhx\" (UniqueName: \"kubernetes.io/projected/a2c4df61-0316-4955-97dc-84b857a09e45-kube-api-access-q7nhx\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hm82z\" (UID: \"a2c4df61-0316-4955-97dc-84b857a09e45\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hm82z" Jul 28 22:16:06.783454 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:06.783440 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2c4df61-0316-4955-97dc-84b857a09e45-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hm82z\" (UID: \"a2c4df61-0316-4955-97dc-84b857a09e45\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hm82z" Jul 28 22:16:06.883970 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:06.883939 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a2c4df61-0316-4955-97dc-84b857a09e45-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hm82z\" (UID: \"a2c4df61-0316-4955-97dc-84b857a09e45\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hm82z" Jul 28 22:16:06.884148 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:06.884006 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7nhx\" (UniqueName: \"kubernetes.io/projected/a2c4df61-0316-4955-97dc-84b857a09e45-kube-api-access-q7nhx\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hm82z\" (UID: \"a2c4df61-0316-4955-97dc-84b857a09e45\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hm82z" Jul 28 22:16:06.884148 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:06.884040 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2c4df61-0316-4955-97dc-84b857a09e45-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hm82z\" (UID: \"a2c4df61-0316-4955-97dc-84b857a09e45\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hm82z" Jul 28 22:16:06.884373 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:06.884351 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a2c4df61-0316-4955-97dc-84b857a09e45-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hm82z\" (UID: \"a2c4df61-0316-4955-97dc-84b857a09e45\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hm82z" Jul 28 22:16:06.884436 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:06.884374 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2c4df61-0316-4955-97dc-84b857a09e45-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hm82z\" (UID: \"a2c4df61-0316-4955-97dc-84b857a09e45\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hm82z" Jul 28 22:16:06.892320 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:06.892295 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7nhx\" (UniqueName: \"kubernetes.io/projected/a2c4df61-0316-4955-97dc-84b857a09e45-kube-api-access-q7nhx\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hm82z\" (UID: \"a2c4df61-0316-4955-97dc-84b857a09e45\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hm82z" Jul 28 22:16:07.020019 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:07.019993 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hm82z" Jul 28 22:16:07.149593 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:07.149571 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hm82z"] Jul 28 22:16:07.151175 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:16:07.151153 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2c4df61_0316_4955_97dc_84b857a09e45.slice/crio-4c4f83af726575f5aa651bbc932286e97f67120c8c91105efa535b146774d5e4 WatchSource:0}: Error finding container 4c4f83af726575f5aa651bbc932286e97f67120c8c91105efa535b146774d5e4: Status 404 returned error can't find the container with id 4c4f83af726575f5aa651bbc932286e97f67120c8c91105efa535b146774d5e4 Jul 28 22:16:07.267072 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:07.267042 2553 generic.go:356] "Generic (PLEG): container finished" podID="a2c4df61-0316-4955-97dc-84b857a09e45" containerID="cb63d14f1728225081404a9354b038afb706460679644632206c4444f3e08312" exitCode=0 Jul 28 22:16:07.267208 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:07.267119 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hm82z" event={"ID":"a2c4df61-0316-4955-97dc-84b857a09e45","Type":"ContainerDied","Data":"cb63d14f1728225081404a9354b038afb706460679644632206c4444f3e08312"} Jul 28 22:16:07.267208 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:07.267170 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hm82z" event={"ID":"a2c4df61-0316-4955-97dc-84b857a09e45","Type":"ContainerStarted","Data":"4c4f83af726575f5aa651bbc932286e97f67120c8c91105efa535b146774d5e4"} Jul 28 22:16:07.269079 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:07.269059 2553 generic.go:356] "Generic (PLEG): container finished" podID="39ea32e0-a0d7-434f-832d-3a934734b409" containerID="bfde7fe8b68a74ea2b2fecc2272cd5619359d0e16e5453f79b83dadc76fe9c56" exitCode=0 Jul 28 22:16:07.269200 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:07.269116 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1frrnl" event={"ID":"39ea32e0-a0d7-434f-832d-3a934734b409","Type":"ContainerDied","Data":"bfde7fe8b68a74ea2b2fecc2272cd5619359d0e16e5453f79b83dadc76fe9c56"} Jul 28 22:16:07.270990 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:07.270972 2553 generic.go:356] "Generic (PLEG): container finished" podID="9ecff602-ff0f-4275-92e1-24ed7c903af2" containerID="a79f45b793ef1a2a867cc4e68ca2d85fb72c07f9fef4c852c5a5998b3b296973" exitCode=0 Jul 28 22:16:07.271067 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:07.271029 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p4rp6" event={"ID":"9ecff602-ff0f-4275-92e1-24ed7c903af2","Type":"ContainerDied","Data":"a79f45b793ef1a2a867cc4e68ca2d85fb72c07f9fef4c852c5a5998b3b296973"} Jul 28 22:16:07.271067 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:07.271058 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p4rp6" event={"ID":"9ecff602-ff0f-4275-92e1-24ed7c903af2","Type":"ContainerStarted","Data":"a096484080ee219d5e18cc493b03c4a2510cef5d12b11286d1aef60979ce83cb"} Jul 28 22:16:07.313880 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:07.313850 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sbp9r"] Jul 28 22:16:07.318610 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:07.318513 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sbp9r" Jul 28 22:16:07.326207 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:07.326187 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sbp9r"] Jul 28 22:16:07.388221 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:07.388187 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f8736b6-aec3-4649-9ea2-a7425f6cc890-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sbp9r\" (UID: \"6f8736b6-aec3-4649-9ea2-a7425f6cc890\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sbp9r" Jul 28 22:16:07.388364 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:07.388228 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glp29\" (UniqueName: \"kubernetes.io/projected/6f8736b6-aec3-4649-9ea2-a7425f6cc890-kube-api-access-glp29\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sbp9r\" (UID: \"6f8736b6-aec3-4649-9ea2-a7425f6cc890\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sbp9r" Jul 28 22:16:07.388364 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:07.388305 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f8736b6-aec3-4649-9ea2-a7425f6cc890-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sbp9r\" (UID: \"6f8736b6-aec3-4649-9ea2-a7425f6cc890\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sbp9r" Jul 28 22:16:07.489248 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:07.489217 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f8736b6-aec3-4649-9ea2-a7425f6cc890-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sbp9r\" (UID: \"6f8736b6-aec3-4649-9ea2-a7425f6cc890\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sbp9r" Jul 28 22:16:07.489608 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:07.489266 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glp29\" (UniqueName: \"kubernetes.io/projected/6f8736b6-aec3-4649-9ea2-a7425f6cc890-kube-api-access-glp29\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sbp9r\" (UID: \"6f8736b6-aec3-4649-9ea2-a7425f6cc890\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sbp9r" Jul 28 22:16:07.489608 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:07.489316 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f8736b6-aec3-4649-9ea2-a7425f6cc890-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sbp9r\" (UID: \"6f8736b6-aec3-4649-9ea2-a7425f6cc890\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sbp9r" Jul 28 22:16:07.489694 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:07.489649 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f8736b6-aec3-4649-9ea2-a7425f6cc890-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sbp9r\" (UID: \"6f8736b6-aec3-4649-9ea2-a7425f6cc890\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sbp9r" Jul 28 22:16:07.489694 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:07.489686 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f8736b6-aec3-4649-9ea2-a7425f6cc890-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sbp9r\" (UID: \"6f8736b6-aec3-4649-9ea2-a7425f6cc890\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sbp9r" Jul 28 22:16:07.497188 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:07.497132 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glp29\" (UniqueName: \"kubernetes.io/projected/6f8736b6-aec3-4649-9ea2-a7425f6cc890-kube-api-access-glp29\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sbp9r\" (UID: \"6f8736b6-aec3-4649-9ea2-a7425f6cc890\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sbp9r" Jul 28 22:16:07.636971 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:07.636927 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sbp9r" Jul 28 22:16:07.766335 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:07.766312 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sbp9r"] Jul 28 22:16:07.768357 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:16:07.768329 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f8736b6_aec3_4649_9ea2_a7425f6cc890.slice/crio-44f1290e607dc7faaf7c5754686773d0f87260de7536d58e99b6712ad0290e00 WatchSource:0}: Error finding container 44f1290e607dc7faaf7c5754686773d0f87260de7536d58e99b6712ad0290e00: Status 404 returned error can't find the container with id 44f1290e607dc7faaf7c5754686773d0f87260de7536d58e99b6712ad0290e00 Jul 28 22:16:08.277751 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:08.277652 2553 generic.go:356] "Generic (PLEG): container finished" podID="a2c4df61-0316-4955-97dc-84b857a09e45" containerID="52fa84ef03534ed9f7a7f0f810121d45fec557f312a53c1349aa4eb465587eb4" exitCode=0 Jul 28 22:16:08.277751 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:08.277741 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hm82z" event={"ID":"a2c4df61-0316-4955-97dc-84b857a09e45","Type":"ContainerDied","Data":"52fa84ef03534ed9f7a7f0f810121d45fec557f312a53c1349aa4eb465587eb4"} Jul 28 22:16:08.279390 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:08.279321 2553 generic.go:356] "Generic (PLEG): container finished" podID="6f8736b6-aec3-4649-9ea2-a7425f6cc890" containerID="498227e4f68d446244f78c6c4c8c42596b795533aff6eb0b3b59e319ea3b03ab" exitCode=0 Jul 28 22:16:08.279467 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:08.279409 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sbp9r" event={"ID":"6f8736b6-aec3-4649-9ea2-a7425f6cc890","Type":"ContainerDied","Data":"498227e4f68d446244f78c6c4c8c42596b795533aff6eb0b3b59e319ea3b03ab"} Jul 28 22:16:08.279467 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:08.279451 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sbp9r" event={"ID":"6f8736b6-aec3-4649-9ea2-a7425f6cc890","Type":"ContainerStarted","Data":"44f1290e607dc7faaf7c5754686773d0f87260de7536d58e99b6712ad0290e00"} Jul 28 22:16:08.281598 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:08.281578 2553 generic.go:356] "Generic (PLEG): container finished" podID="39ea32e0-a0d7-434f-832d-3a934734b409" containerID="b6d826d6071da9a27e5a4aaa51d76f73c8b3250a91f3a256845492d6f40d54eb" exitCode=0 Jul 28 22:16:08.281675 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:08.281653 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1frrnl" event={"ID":"39ea32e0-a0d7-434f-832d-3a934734b409","Type":"ContainerDied","Data":"b6d826d6071da9a27e5a4aaa51d76f73c8b3250a91f3a256845492d6f40d54eb"} Jul 28 22:16:08.283415 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:08.283392 2553 generic.go:356] "Generic (PLEG): container finished" podID="9ecff602-ff0f-4275-92e1-24ed7c903af2" containerID="2ed043f7b31b6216a7fff2a07292e524df26e1a4eacad6a21c87466225cc5d39" exitCode=0 Jul 28 22:16:08.283513 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:08.283461 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p4rp6" event={"ID":"9ecff602-ff0f-4275-92e1-24ed7c903af2","Type":"ContainerDied","Data":"2ed043f7b31b6216a7fff2a07292e524df26e1a4eacad6a21c87466225cc5d39"} Jul 28 22:16:09.290571 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:09.290542 2553 generic.go:356] "Generic (PLEG): container finished" podID="9ecff602-ff0f-4275-92e1-24ed7c903af2" containerID="60d980bd30599b125f131035d268bc14f9270f8b3289137c3ec90ce151fb2322" exitCode=0 Jul 28 22:16:09.291025 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:09.290621 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p4rp6" event={"ID":"9ecff602-ff0f-4275-92e1-24ed7c903af2","Type":"ContainerDied","Data":"60d980bd30599b125f131035d268bc14f9270f8b3289137c3ec90ce151fb2322"} Jul 28 22:16:09.292679 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:09.292655 2553 generic.go:356] "Generic (PLEG): container finished" podID="a2c4df61-0316-4955-97dc-84b857a09e45" containerID="649cd97e1855e2f94d61e422b09ad531db8ce78ebdbda903a99b7633bab4ce0e" exitCode=0 Jul 28 22:16:09.292820 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:09.292738 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hm82z" event={"ID":"a2c4df61-0316-4955-97dc-84b857a09e45","Type":"ContainerDied","Data":"649cd97e1855e2f94d61e422b09ad531db8ce78ebdbda903a99b7633bab4ce0e"} Jul 28 22:16:09.294455 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:09.294432 2553 generic.go:356] "Generic (PLEG): container finished" podID="6f8736b6-aec3-4649-9ea2-a7425f6cc890" containerID="761d8de71f2d4d48169d7bed2a1e62862a2db0dc86a2cc3bfc4da7e2e724d5b3" exitCode=0 Jul 28 22:16:09.294561 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:09.294503 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sbp9r" event={"ID":"6f8736b6-aec3-4649-9ea2-a7425f6cc890","Type":"ContainerDied","Data":"761d8de71f2d4d48169d7bed2a1e62862a2db0dc86a2cc3bfc4da7e2e724d5b3"} Jul 28 22:16:09.429016 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:09.428945 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1frrnl" Jul 28 22:16:09.505100 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:09.505064 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/39ea32e0-a0d7-434f-832d-3a934734b409-bundle\") pod \"39ea32e0-a0d7-434f-832d-3a934734b409\" (UID: \"39ea32e0-a0d7-434f-832d-3a934734b409\") " Jul 28 22:16:09.505100 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:09.505106 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/39ea32e0-a0d7-434f-832d-3a934734b409-util\") pod \"39ea32e0-a0d7-434f-832d-3a934734b409\" (UID: \"39ea32e0-a0d7-434f-832d-3a934734b409\") " Jul 28 22:16:09.505308 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:09.505134 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5csfg\" (UniqueName: \"kubernetes.io/projected/39ea32e0-a0d7-434f-832d-3a934734b409-kube-api-access-5csfg\") pod \"39ea32e0-a0d7-434f-832d-3a934734b409\" (UID: \"39ea32e0-a0d7-434f-832d-3a934734b409\") " Jul 28 22:16:09.505685 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:09.505656 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39ea32e0-a0d7-434f-832d-3a934734b409-bundle" (OuterVolumeSpecName: "bundle") pod "39ea32e0-a0d7-434f-832d-3a934734b409" (UID: "39ea32e0-a0d7-434f-832d-3a934734b409"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:16:09.507522 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:09.507501 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39ea32e0-a0d7-434f-832d-3a934734b409-kube-api-access-5csfg" (OuterVolumeSpecName: "kube-api-access-5csfg") pod "39ea32e0-a0d7-434f-832d-3a934734b409" (UID: "39ea32e0-a0d7-434f-832d-3a934734b409"). InnerVolumeSpecName "kube-api-access-5csfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:16:09.510425 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:09.510407 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39ea32e0-a0d7-434f-832d-3a934734b409-util" (OuterVolumeSpecName: "util") pod "39ea32e0-a0d7-434f-832d-3a934734b409" (UID: "39ea32e0-a0d7-434f-832d-3a934734b409"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:16:09.606419 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:09.606353 2553 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/39ea32e0-a0d7-434f-832d-3a934734b409-bundle\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:16:09.606419 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:09.606375 2553 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/39ea32e0-a0d7-434f-832d-3a934734b409-util\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:16:09.606419 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:09.606384 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5csfg\" (UniqueName: \"kubernetes.io/projected/39ea32e0-a0d7-434f-832d-3a934734b409-kube-api-access-5csfg\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:16:10.300925 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:10.300895 2553 generic.go:356] "Generic (PLEG): container finished" podID="6f8736b6-aec3-4649-9ea2-a7425f6cc890" containerID="82a3a09d80c6b44703d070993b5ae7423f31de1ede464b3601ec1340a0c3ea06" exitCode=0 Jul 28 22:16:10.301351 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:10.300977 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sbp9r" event={"ID":"6f8736b6-aec3-4649-9ea2-a7425f6cc890","Type":"ContainerDied","Data":"82a3a09d80c6b44703d070993b5ae7423f31de1ede464b3601ec1340a0c3ea06"} Jul 28 22:16:10.302833 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:10.302816 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1frrnl" Jul 28 22:16:10.303097 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:10.302856 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1frrnl" event={"ID":"39ea32e0-a0d7-434f-832d-3a934734b409","Type":"ContainerDied","Data":"39407de9d7fcfea18b6d3c0ceed5287a5d5875247d0de0baebe0fe59335ba265"} Jul 28 22:16:10.303097 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:10.302898 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39407de9d7fcfea18b6d3c0ceed5287a5d5875247d0de0baebe0fe59335ba265" Jul 28 22:16:10.471254 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:10.471219 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p4rp6" Jul 28 22:16:10.473283 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:10.473268 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hm82z" Jul 28 22:16:10.513126 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:10.513099 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a2c4df61-0316-4955-97dc-84b857a09e45-util\") pod \"a2c4df61-0316-4955-97dc-84b857a09e45\" (UID: \"a2c4df61-0316-4955-97dc-84b857a09e45\") " Jul 28 22:16:10.513276 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:10.513151 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q7nhx\" (UniqueName: \"kubernetes.io/projected/a2c4df61-0316-4955-97dc-84b857a09e45-kube-api-access-q7nhx\") pod \"a2c4df61-0316-4955-97dc-84b857a09e45\" (UID: \"a2c4df61-0316-4955-97dc-84b857a09e45\") " Jul 28 22:16:10.513276 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:10.513185 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9ecff602-ff0f-4275-92e1-24ed7c903af2-bundle\") pod \"9ecff602-ff0f-4275-92e1-24ed7c903af2\" (UID: \"9ecff602-ff0f-4275-92e1-24ed7c903af2\") " Jul 28 22:16:10.513276 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:10.513234 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2c4df61-0316-4955-97dc-84b857a09e45-bundle\") pod \"a2c4df61-0316-4955-97dc-84b857a09e45\" (UID: \"a2c4df61-0316-4955-97dc-84b857a09e45\") " Jul 28 22:16:10.513276 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:10.513268 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9ecff602-ff0f-4275-92e1-24ed7c903af2-util\") pod \"9ecff602-ff0f-4275-92e1-24ed7c903af2\" (UID: \"9ecff602-ff0f-4275-92e1-24ed7c903af2\") " Jul 28 22:16:10.513483 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:10.513340 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kpfst\" (UniqueName: \"kubernetes.io/projected/9ecff602-ff0f-4275-92e1-24ed7c903af2-kube-api-access-kpfst\") pod \"9ecff602-ff0f-4275-92e1-24ed7c903af2\" (UID: \"9ecff602-ff0f-4275-92e1-24ed7c903af2\") " Jul 28 22:16:10.513734 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:10.513679 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2c4df61-0316-4955-97dc-84b857a09e45-bundle" (OuterVolumeSpecName: "bundle") pod "a2c4df61-0316-4955-97dc-84b857a09e45" (UID: "a2c4df61-0316-4955-97dc-84b857a09e45"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:16:10.514127 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:10.514073 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ecff602-ff0f-4275-92e1-24ed7c903af2-bundle" (OuterVolumeSpecName: "bundle") pod "9ecff602-ff0f-4275-92e1-24ed7c903af2" (UID: "9ecff602-ff0f-4275-92e1-24ed7c903af2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:16:10.516115 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:10.516091 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2c4df61-0316-4955-97dc-84b857a09e45-kube-api-access-q7nhx" (OuterVolumeSpecName: "kube-api-access-q7nhx") pod "a2c4df61-0316-4955-97dc-84b857a09e45" (UID: "a2c4df61-0316-4955-97dc-84b857a09e45"). InnerVolumeSpecName "kube-api-access-q7nhx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:16:10.516278 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:10.516254 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ecff602-ff0f-4275-92e1-24ed7c903af2-kube-api-access-kpfst" (OuterVolumeSpecName: "kube-api-access-kpfst") pod "9ecff602-ff0f-4275-92e1-24ed7c903af2" (UID: "9ecff602-ff0f-4275-92e1-24ed7c903af2"). InnerVolumeSpecName "kube-api-access-kpfst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:16:10.519736 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:10.519717 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2c4df61-0316-4955-97dc-84b857a09e45-util" (OuterVolumeSpecName: "util") pod "a2c4df61-0316-4955-97dc-84b857a09e45" (UID: "a2c4df61-0316-4955-97dc-84b857a09e45"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:16:10.520047 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:10.520029 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ecff602-ff0f-4275-92e1-24ed7c903af2-util" (OuterVolumeSpecName: "util") pod "9ecff602-ff0f-4275-92e1-24ed7c903af2" (UID: "9ecff602-ff0f-4275-92e1-24ed7c903af2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:16:10.615095 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:10.615039 2553 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a2c4df61-0316-4955-97dc-84b857a09e45-util\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:16:10.615095 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:10.615061 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7nhx\" (UniqueName: \"kubernetes.io/projected/a2c4df61-0316-4955-97dc-84b857a09e45-kube-api-access-q7nhx\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:16:10.615095 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:10.615071 2553 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9ecff602-ff0f-4275-92e1-24ed7c903af2-bundle\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:16:10.615095 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:10.615079 2553 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2c4df61-0316-4955-97dc-84b857a09e45-bundle\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:16:10.615095 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:10.615086 2553 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9ecff602-ff0f-4275-92e1-24ed7c903af2-util\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:16:10.615095 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:10.615093 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpfst\" (UniqueName: \"kubernetes.io/projected/9ecff602-ff0f-4275-92e1-24ed7c903af2-kube-api-access-kpfst\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:16:11.309354 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:11.309327 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p4rp6" Jul 28 22:16:11.309354 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:11.309327 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p4rp6" event={"ID":"9ecff602-ff0f-4275-92e1-24ed7c903af2","Type":"ContainerDied","Data":"a096484080ee219d5e18cc493b03c4a2510cef5d12b11286d1aef60979ce83cb"} Jul 28 22:16:11.310032 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:11.309408 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a096484080ee219d5e18cc493b03c4a2510cef5d12b11286d1aef60979ce83cb" Jul 28 22:16:11.311465 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:11.311435 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hm82z" event={"ID":"a2c4df61-0316-4955-97dc-84b857a09e45","Type":"ContainerDied","Data":"4c4f83af726575f5aa651bbc932286e97f67120c8c91105efa535b146774d5e4"} Jul 28 22:16:11.311581 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:11.311484 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c4f83af726575f5aa651bbc932286e97f67120c8c91105efa535b146774d5e4" Jul 28 22:16:11.311621 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:11.311598 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hm82z" Jul 28 22:16:11.322088 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:16:11.322055 2553 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2c4df61_0316_4955_97dc_84b857a09e45.slice\": RecentStats: unable to find data in memory cache]" Jul 28 22:16:11.444514 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:11.444486 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sbp9r" Jul 28 22:16:11.522994 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:11.522971 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-glp29\" (UniqueName: \"kubernetes.io/projected/6f8736b6-aec3-4649-9ea2-a7425f6cc890-kube-api-access-glp29\") pod \"6f8736b6-aec3-4649-9ea2-a7425f6cc890\" (UID: \"6f8736b6-aec3-4649-9ea2-a7425f6cc890\") " Jul 28 22:16:11.523161 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:11.523028 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f8736b6-aec3-4649-9ea2-a7425f6cc890-bundle\") pod \"6f8736b6-aec3-4649-9ea2-a7425f6cc890\" (UID: \"6f8736b6-aec3-4649-9ea2-a7425f6cc890\") " Jul 28 22:16:11.523161 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:11.523100 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f8736b6-aec3-4649-9ea2-a7425f6cc890-util\") pod \"6f8736b6-aec3-4649-9ea2-a7425f6cc890\" (UID: \"6f8736b6-aec3-4649-9ea2-a7425f6cc890\") " Jul 28 22:16:11.523533 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:11.523507 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f8736b6-aec3-4649-9ea2-a7425f6cc890-bundle" (OuterVolumeSpecName: "bundle") pod "6f8736b6-aec3-4649-9ea2-a7425f6cc890" (UID: "6f8736b6-aec3-4649-9ea2-a7425f6cc890"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:16:11.525397 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:11.525370 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f8736b6-aec3-4649-9ea2-a7425f6cc890-kube-api-access-glp29" (OuterVolumeSpecName: "kube-api-access-glp29") pod "6f8736b6-aec3-4649-9ea2-a7425f6cc890" (UID: "6f8736b6-aec3-4649-9ea2-a7425f6cc890"). InnerVolumeSpecName "kube-api-access-glp29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:16:11.531562 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:11.531534 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f8736b6-aec3-4649-9ea2-a7425f6cc890-util" (OuterVolumeSpecName: "util") pod "6f8736b6-aec3-4649-9ea2-a7425f6cc890" (UID: "6f8736b6-aec3-4649-9ea2-a7425f6cc890"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:16:11.624044 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:11.623984 2553 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f8736b6-aec3-4649-9ea2-a7425f6cc890-bundle\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:16:11.624044 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:11.624003 2553 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f8736b6-aec3-4649-9ea2-a7425f6cc890-util\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:16:11.624044 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:11.624013 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glp29\" (UniqueName: \"kubernetes.io/projected/6f8736b6-aec3-4649-9ea2-a7425f6cc890-kube-api-access-glp29\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:16:12.318443 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:12.318417 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sbp9r" Jul 28 22:16:12.318855 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:12.318420 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sbp9r" event={"ID":"6f8736b6-aec3-4649-9ea2-a7425f6cc890","Type":"ContainerDied","Data":"44f1290e607dc7faaf7c5754686773d0f87260de7536d58e99b6712ad0290e00"} Jul 28 22:16:12.318855 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:12.318533 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44f1290e607dc7faaf7c5754686773d0f87260de7536d58e99b6712ad0290e00" Jul 28 22:16:38.438238 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:16:38.438201 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6dd6575cd5-zxwkt"] Jul 28 22:17:03.460380 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:03.460333 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6dd6575cd5-zxwkt" podUID="e4e4da99-d77a-4219-b075-d5d3f4715547" containerName="console" containerID="cri-o://cdcabeedacf7f88bd36d5c42e10bbb5e9b9cf9822aafc4ea1be5619d4520f4bb" gracePeriod=15 Jul 28 22:17:03.716024 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:03.716001 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dd6575cd5-zxwkt" Jul 28 22:17:03.753362 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:03.753331 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e4e4da99-d77a-4219-b075-d5d3f4715547-console-oauth-config\") pod \"e4e4da99-d77a-4219-b075-d5d3f4715547\" (UID: \"e4e4da99-d77a-4219-b075-d5d3f4715547\") " Jul 28 22:17:03.753555 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:03.753373 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e4e4da99-d77a-4219-b075-d5d3f4715547-oauth-serving-cert\") pod \"e4e4da99-d77a-4219-b075-d5d3f4715547\" (UID: \"e4e4da99-d77a-4219-b075-d5d3f4715547\") " Jul 28 22:17:03.753555 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:03.753411 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e4e4da99-d77a-4219-b075-d5d3f4715547-service-ca\") pod \"e4e4da99-d77a-4219-b075-d5d3f4715547\" (UID: \"e4e4da99-d77a-4219-b075-d5d3f4715547\") " Jul 28 22:17:03.753555 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:03.753433 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e4e4da99-d77a-4219-b075-d5d3f4715547-console-config\") pod \"e4e4da99-d77a-4219-b075-d5d3f4715547\" (UID: \"e4e4da99-d77a-4219-b075-d5d3f4715547\") " Jul 28 22:17:03.753555 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:03.753451 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4e4da99-d77a-4219-b075-d5d3f4715547-trusted-ca-bundle\") pod \"e4e4da99-d77a-4219-b075-d5d3f4715547\" (UID: \"e4e4da99-d77a-4219-b075-d5d3f4715547\") " Jul 28 22:17:03.753555 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:03.753473 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tvgt9\" (UniqueName: \"kubernetes.io/projected/e4e4da99-d77a-4219-b075-d5d3f4715547-kube-api-access-tvgt9\") pod \"e4e4da99-d77a-4219-b075-d5d3f4715547\" (UID: \"e4e4da99-d77a-4219-b075-d5d3f4715547\") " Jul 28 22:17:03.753555 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:03.753511 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e4e4da99-d77a-4219-b075-d5d3f4715547-console-serving-cert\") pod \"e4e4da99-d77a-4219-b075-d5d3f4715547\" (UID: \"e4e4da99-d77a-4219-b075-d5d3f4715547\") " Jul 28 22:17:03.753947 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:03.753847 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4e4da99-d77a-4219-b075-d5d3f4715547-console-config" (OuterVolumeSpecName: "console-config") pod "e4e4da99-d77a-4219-b075-d5d3f4715547" (UID: "e4e4da99-d77a-4219-b075-d5d3f4715547"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 22:17:03.753947 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:03.753820 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4e4da99-d77a-4219-b075-d5d3f4715547-service-ca" (OuterVolumeSpecName: "service-ca") pod "e4e4da99-d77a-4219-b075-d5d3f4715547" (UID: "e4e4da99-d77a-4219-b075-d5d3f4715547"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 22:17:03.753947 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:03.753932 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4e4da99-d77a-4219-b075-d5d3f4715547-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e4e4da99-d77a-4219-b075-d5d3f4715547" (UID: "e4e4da99-d77a-4219-b075-d5d3f4715547"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 22:17:03.754117 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:03.754059 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4e4da99-d77a-4219-b075-d5d3f4715547-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "e4e4da99-d77a-4219-b075-d5d3f4715547" (UID: "e4e4da99-d77a-4219-b075-d5d3f4715547"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 22:17:03.755827 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:03.755793 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4e4da99-d77a-4219-b075-d5d3f4715547-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e4e4da99-d77a-4219-b075-d5d3f4715547" (UID: "e4e4da99-d77a-4219-b075-d5d3f4715547"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 22:17:03.755943 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:03.755872 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4e4da99-d77a-4219-b075-d5d3f4715547-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e4e4da99-d77a-4219-b075-d5d3f4715547" (UID: "e4e4da99-d77a-4219-b075-d5d3f4715547"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 22:17:03.756075 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:03.756054 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4e4da99-d77a-4219-b075-d5d3f4715547-kube-api-access-tvgt9" (OuterVolumeSpecName: "kube-api-access-tvgt9") pod "e4e4da99-d77a-4219-b075-d5d3f4715547" (UID: "e4e4da99-d77a-4219-b075-d5d3f4715547"). InnerVolumeSpecName "kube-api-access-tvgt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:17:03.854117 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:03.854082 2553 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e4e4da99-d77a-4219-b075-d5d3f4715547-console-config\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:17:03.854117 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:03.854109 2553 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4e4da99-d77a-4219-b075-d5d3f4715547-trusted-ca-bundle\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:17:03.854117 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:03.854118 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvgt9\" (UniqueName: \"kubernetes.io/projected/e4e4da99-d77a-4219-b075-d5d3f4715547-kube-api-access-tvgt9\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:17:03.854117 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:03.854128 2553 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e4e4da99-d77a-4219-b075-d5d3f4715547-console-serving-cert\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:17:03.854428 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:03.854138 2553 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e4e4da99-d77a-4219-b075-d5d3f4715547-console-oauth-config\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:17:03.854428 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:03.854146 2553 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e4e4da99-d77a-4219-b075-d5d3f4715547-oauth-serving-cert\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:17:03.854428 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:03.854153 2553 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e4e4da99-d77a-4219-b075-d5d3f4715547-service-ca\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:17:04.573321 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:04.573283 2553 generic.go:356] "Generic (PLEG): container finished" podID="e4e4da99-d77a-4219-b075-d5d3f4715547" containerID="cdcabeedacf7f88bd36d5c42e10bbb5e9b9cf9822aafc4ea1be5619d4520f4bb" exitCode=0 Jul 28 22:17:04.573797 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:04.573335 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dd6575cd5-zxwkt" Jul 28 22:17:04.573797 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:04.573370 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dd6575cd5-zxwkt" event={"ID":"e4e4da99-d77a-4219-b075-d5d3f4715547","Type":"ContainerDied","Data":"cdcabeedacf7f88bd36d5c42e10bbb5e9b9cf9822aafc4ea1be5619d4520f4bb"} Jul 28 22:17:04.573797 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:04.573427 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dd6575cd5-zxwkt" event={"ID":"e4e4da99-d77a-4219-b075-d5d3f4715547","Type":"ContainerDied","Data":"80738b97abac773d691304625561c52520bbcf2680d44ad4d2aedf11347c318e"} Jul 28 22:17:04.573797 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:04.573449 2553 scope.go:117] "RemoveContainer" containerID="cdcabeedacf7f88bd36d5c42e10bbb5e9b9cf9822aafc4ea1be5619d4520f4bb" Jul 28 22:17:04.582203 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:04.582181 2553 scope.go:117] "RemoveContainer" containerID="cdcabeedacf7f88bd36d5c42e10bbb5e9b9cf9822aafc4ea1be5619d4520f4bb" Jul 28 22:17:04.582428 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:17:04.582410 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdcabeedacf7f88bd36d5c42e10bbb5e9b9cf9822aafc4ea1be5619d4520f4bb\": container with ID starting with cdcabeedacf7f88bd36d5c42e10bbb5e9b9cf9822aafc4ea1be5619d4520f4bb not found: ID does not exist" containerID="cdcabeedacf7f88bd36d5c42e10bbb5e9b9cf9822aafc4ea1be5619d4520f4bb" Jul 28 22:17:04.582490 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:04.582436 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdcabeedacf7f88bd36d5c42e10bbb5e9b9cf9822aafc4ea1be5619d4520f4bb"} err="failed to get container status \"cdcabeedacf7f88bd36d5c42e10bbb5e9b9cf9822aafc4ea1be5619d4520f4bb\": rpc error: code = NotFound desc = could not find container \"cdcabeedacf7f88bd36d5c42e10bbb5e9b9cf9822aafc4ea1be5619d4520f4bb\": container with ID starting with cdcabeedacf7f88bd36d5c42e10bbb5e9b9cf9822aafc4ea1be5619d4520f4bb not found: ID does not exist" Jul 28 22:17:04.592215 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:04.592195 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6dd6575cd5-zxwkt"] Jul 28 22:17:04.598827 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:04.598801 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6dd6575cd5-zxwkt"] Jul 28 22:17:06.404206 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:06.404179 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4e4da99-d77a-4219-b075-d5d3f4715547" path="/var/lib/kubelet/pods/e4e4da99-d77a-4219-b075-d5d3f4715547/volumes" Jul 28 22:17:25.925116 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:25.925079 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfrsf"] Jul 28 22:17:25.925594 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:25.925447 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f8736b6-aec3-4649-9ea2-a7425f6cc890" containerName="pull" Jul 28 22:17:25.925594 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:25.925463 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f8736b6-aec3-4649-9ea2-a7425f6cc890" containerName="pull" Jul 28 22:17:25.925594 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:25.925477 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4e4da99-d77a-4219-b075-d5d3f4715547" containerName="console" Jul 28 22:17:25.925594 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:25.925485 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4e4da99-d77a-4219-b075-d5d3f4715547" containerName="console" Jul 28 22:17:25.925594 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:25.925504 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2c4df61-0316-4955-97dc-84b857a09e45" containerName="pull" Jul 28 22:17:25.925594 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:25.925511 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2c4df61-0316-4955-97dc-84b857a09e45" containerName="pull" Jul 28 22:17:25.925594 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:25.925518 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39ea32e0-a0d7-434f-832d-3a934734b409" containerName="extract" Jul 28 22:17:25.925594 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:25.925522 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="39ea32e0-a0d7-434f-832d-3a934734b409" containerName="extract" Jul 28 22:17:25.925594 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:25.925528 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ecff602-ff0f-4275-92e1-24ed7c903af2" containerName="extract" Jul 28 22:17:25.925594 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:25.925533 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ecff602-ff0f-4275-92e1-24ed7c903af2" containerName="extract" Jul 28 22:17:25.925594 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:25.925541 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39ea32e0-a0d7-434f-832d-3a934734b409" containerName="util" Jul 28 22:17:25.925594 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:25.925548 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="39ea32e0-a0d7-434f-832d-3a934734b409" containerName="util" Jul 28 22:17:25.925594 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:25.925556 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39ea32e0-a0d7-434f-832d-3a934734b409" containerName="pull" Jul 28 22:17:25.925594 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:25.925562 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="39ea32e0-a0d7-434f-832d-3a934734b409" containerName="pull" Jul 28 22:17:25.925594 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:25.925572 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ecff602-ff0f-4275-92e1-24ed7c903af2" containerName="util" Jul 28 22:17:25.925594 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:25.925581 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ecff602-ff0f-4275-92e1-24ed7c903af2" containerName="util" Jul 28 22:17:25.925594 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:25.925593 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2c4df61-0316-4955-97dc-84b857a09e45" containerName="extract" Jul 28 22:17:25.925594 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:25.925601 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2c4df61-0316-4955-97dc-84b857a09e45" containerName="extract" Jul 28 22:17:25.926179 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:25.925615 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2c4df61-0316-4955-97dc-84b857a09e45" containerName="util" Jul 28 22:17:25.926179 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:25.925621 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2c4df61-0316-4955-97dc-84b857a09e45" containerName="util" Jul 28 22:17:25.926179 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:25.925626 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ecff602-ff0f-4275-92e1-24ed7c903af2" containerName="pull" Jul 28 22:17:25.926179 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:25.925632 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ecff602-ff0f-4275-92e1-24ed7c903af2" containerName="pull" Jul 28 22:17:25.926179 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:25.925637 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f8736b6-aec3-4649-9ea2-a7425f6cc890" containerName="extract" Jul 28 22:17:25.926179 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:25.925643 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f8736b6-aec3-4649-9ea2-a7425f6cc890" containerName="extract" Jul 28 22:17:25.926179 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:25.925654 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f8736b6-aec3-4649-9ea2-a7425f6cc890" containerName="util" Jul 28 22:17:25.926179 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:25.925661 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f8736b6-aec3-4649-9ea2-a7425f6cc890" containerName="util" Jul 28 22:17:25.926179 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:25.925753 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f8736b6-aec3-4649-9ea2-a7425f6cc890" containerName="extract" Jul 28 22:17:25.926179 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:25.925768 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4e4da99-d77a-4219-b075-d5d3f4715547" containerName="console" Jul 28 22:17:25.926179 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:25.925777 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2c4df61-0316-4955-97dc-84b857a09e45" containerName="extract" Jul 28 22:17:25.926179 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:25.925788 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="39ea32e0-a0d7-434f-832d-3a934734b409" containerName="extract" Jul 28 22:17:25.926179 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:25.925799 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ecff602-ff0f-4275-92e1-24ed7c903af2" containerName="extract" Jul 28 22:17:25.930047 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:25.930030 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfrsf" Jul 28 22:17:25.932223 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:25.932202 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-wkw8j\"" Jul 28 22:17:25.942877 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:25.942842 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfrsf"] Jul 28 22:17:26.014311 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:26.014276 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vfzp\" (UniqueName: \"kubernetes.io/projected/e776e265-c83c-42be-be8c-07647513e2ba-kube-api-access-6vfzp\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bfrsf\" (UID: \"e776e265-c83c-42be-be8c-07647513e2ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfrsf" Jul 28 22:17:26.014464 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:26.014313 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e776e265-c83c-42be-be8c-07647513e2ba-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bfrsf\" (UID: \"e776e265-c83c-42be-be8c-07647513e2ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfrsf" Jul 28 22:17:26.014464 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:26.014338 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e776e265-c83c-42be-be8c-07647513e2ba-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bfrsf\" (UID: \"e776e265-c83c-42be-be8c-07647513e2ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfrsf" Jul 28 22:17:26.014464 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:26.014356 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e776e265-c83c-42be-be8c-07647513e2ba-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bfrsf\" (UID: \"e776e265-c83c-42be-be8c-07647513e2ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfrsf" Jul 28 22:17:26.014464 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:26.014419 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e776e265-c83c-42be-be8c-07647513e2ba-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bfrsf\" (UID: \"e776e265-c83c-42be-be8c-07647513e2ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfrsf" Jul 28 22:17:26.014464 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:26.014437 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e776e265-c83c-42be-be8c-07647513e2ba-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bfrsf\" (UID: \"e776e265-c83c-42be-be8c-07647513e2ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfrsf" Jul 28 22:17:26.014631 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:26.014514 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e776e265-c83c-42be-be8c-07647513e2ba-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bfrsf\" (UID: \"e776e265-c83c-42be-be8c-07647513e2ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfrsf" Jul 28 22:17:26.014631 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:26.014547 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e776e265-c83c-42be-be8c-07647513e2ba-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bfrsf\" (UID: \"e776e265-c83c-42be-be8c-07647513e2ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfrsf" Jul 28 22:17:26.014631 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:26.014566 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e776e265-c83c-42be-be8c-07647513e2ba-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bfrsf\" (UID: \"e776e265-c83c-42be-be8c-07647513e2ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfrsf" Jul 28 22:17:26.115911 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:26.115874 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e776e265-c83c-42be-be8c-07647513e2ba-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bfrsf\" (UID: \"e776e265-c83c-42be-be8c-07647513e2ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfrsf" Jul 28 22:17:26.115911 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:26.115915 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e776e265-c83c-42be-be8c-07647513e2ba-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bfrsf\" (UID: \"e776e265-c83c-42be-be8c-07647513e2ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfrsf" Jul 28 22:17:26.116146 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:26.115935 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e776e265-c83c-42be-be8c-07647513e2ba-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bfrsf\" (UID: \"e776e265-c83c-42be-be8c-07647513e2ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfrsf" Jul 28 22:17:26.116146 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:26.115968 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vfzp\" (UniqueName: \"kubernetes.io/projected/e776e265-c83c-42be-be8c-07647513e2ba-kube-api-access-6vfzp\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bfrsf\" (UID: \"e776e265-c83c-42be-be8c-07647513e2ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfrsf" Jul 28 22:17:26.116146 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:26.116000 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e776e265-c83c-42be-be8c-07647513e2ba-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bfrsf\" (UID: \"e776e265-c83c-42be-be8c-07647513e2ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfrsf" Jul 28 22:17:26.116146 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:26.116029 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e776e265-c83c-42be-be8c-07647513e2ba-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bfrsf\" (UID: \"e776e265-c83c-42be-be8c-07647513e2ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfrsf" Jul 28 22:17:26.116146 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:26.116050 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e776e265-c83c-42be-be8c-07647513e2ba-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bfrsf\" (UID: \"e776e265-c83c-42be-be8c-07647513e2ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfrsf" Jul 28 22:17:26.116146 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:26.116085 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e776e265-c83c-42be-be8c-07647513e2ba-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bfrsf\" (UID: \"e776e265-c83c-42be-be8c-07647513e2ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfrsf" Jul 28 22:17:26.116146 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:26.116115 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e776e265-c83c-42be-be8c-07647513e2ba-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bfrsf\" (UID: \"e776e265-c83c-42be-be8c-07647513e2ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfrsf" Jul 28 22:17:26.116518 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:26.116317 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e776e265-c83c-42be-be8c-07647513e2ba-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bfrsf\" (UID: \"e776e265-c83c-42be-be8c-07647513e2ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfrsf" Jul 28 22:17:26.116518 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:26.116379 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e776e265-c83c-42be-be8c-07647513e2ba-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bfrsf\" (UID: \"e776e265-c83c-42be-be8c-07647513e2ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfrsf" Jul 28 22:17:26.116853 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:26.116832 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e776e265-c83c-42be-be8c-07647513e2ba-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bfrsf\" (UID: \"e776e265-c83c-42be-be8c-07647513e2ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfrsf" Jul 28 22:17:26.116972 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:26.116917 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e776e265-c83c-42be-be8c-07647513e2ba-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bfrsf\" (UID: \"e776e265-c83c-42be-be8c-07647513e2ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfrsf" Jul 28 22:17:26.116972 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:26.116923 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e776e265-c83c-42be-be8c-07647513e2ba-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bfrsf\" (UID: \"e776e265-c83c-42be-be8c-07647513e2ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfrsf" Jul 28 22:17:26.118698 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:26.118680 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e776e265-c83c-42be-be8c-07647513e2ba-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bfrsf\" (UID: \"e776e265-c83c-42be-be8c-07647513e2ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfrsf" Jul 28 22:17:26.118874 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:26.118858 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e776e265-c83c-42be-be8c-07647513e2ba-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bfrsf\" (UID: \"e776e265-c83c-42be-be8c-07647513e2ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfrsf" Jul 28 22:17:26.124094 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:26.124068 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e776e265-c83c-42be-be8c-07647513e2ba-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bfrsf\" (UID: \"e776e265-c83c-42be-be8c-07647513e2ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfrsf" Jul 28 22:17:26.124176 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:26.124116 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vfzp\" (UniqueName: \"kubernetes.io/projected/e776e265-c83c-42be-be8c-07647513e2ba-kube-api-access-6vfzp\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bfrsf\" (UID: \"e776e265-c83c-42be-be8c-07647513e2ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfrsf" Jul 28 22:17:26.240193 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:26.240160 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfrsf" Jul 28 22:17:26.383739 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:26.383679 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfrsf"] Jul 28 22:17:26.385469 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:17:26.385444 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode776e265_c83c_42be_be8c_07647513e2ba.slice/crio-cc951b965e76a7f20f4770a699d031fa26c5ad88a3c29b3b126dae39505ff4ba WatchSource:0}: Error finding container cc951b965e76a7f20f4770a699d031fa26c5ad88a3c29b3b126dae39505ff4ba: Status 404 returned error can't find the container with id cc951b965e76a7f20f4770a699d031fa26c5ad88a3c29b3b126dae39505ff4ba Jul 28 22:17:26.387596 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:26.387568 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:17:26.387673 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:26.387630 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:17:26.387673 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:26.387657 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:17:26.685029 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:26.684946 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfrsf" event={"ID":"e776e265-c83c-42be-be8c-07647513e2ba","Type":"ContainerStarted","Data":"2d7f3efd95e65b27dccd3b54ac09a9b5deca8c3218bbddce1393dd47442229df"} Jul 28 22:17:26.685029 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:26.684997 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfrsf" event={"ID":"e776e265-c83c-42be-be8c-07647513e2ba","Type":"ContainerStarted","Data":"cc951b965e76a7f20f4770a699d031fa26c5ad88a3c29b3b126dae39505ff4ba"} Jul 28 22:17:26.707049 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:26.707002 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfrsf" podStartSLOduration=1.7069877629999999 podStartE2EDuration="1.706987763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:17:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:17:26.704971051 +0000 UTC m=+466.792229119" watchObservedRunningTime="2026-07-28 22:17:26.706987763 +0000 UTC m=+466.794245832" Jul 28 22:17:27.240615 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:27.240580 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfrsf" Jul 28 22:17:27.245675 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:27.245650 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfrsf" Jul 28 22:17:27.689910 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:27.689822 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfrsf" Jul 28 22:17:27.690575 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:17:27.690554 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfrsf" Jul 28 22:18:07.585074 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:07.585037 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-fcb85f65f-c2n2w"] Jul 28 22:18:07.588409 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:07.588391 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-fcb85f65f-c2n2w" Jul 28 22:18:07.590822 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:07.590791 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-7k5cv\"" Jul 28 22:18:07.590822 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:07.590805 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 28 22:18:07.590992 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:07.590796 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 28 22:18:07.599109 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:07.599083 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-fcb85f65f-c2n2w"] Jul 28 22:18:07.757150 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:07.757118 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-d6457fd9-7kwsd"] Jul 28 22:18:07.760349 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:07.760333 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d6457fd9-7kwsd" Jul 28 22:18:07.766264 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:07.766240 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/66f3af82-eb7e-45c5-b6b5-20616e528b53-webhook-cert\") pod \"maas-controller-fcb85f65f-c2n2w\" (UID: \"66f3af82-eb7e-45c5-b6b5-20616e528b53\") " pod="opendatahub/maas-controller-fcb85f65f-c2n2w" Jul 28 22:18:07.766264 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:07.766268 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46btn\" (UniqueName: \"kubernetes.io/projected/66f3af82-eb7e-45c5-b6b5-20616e528b53-kube-api-access-46btn\") pod \"maas-controller-fcb85f65f-c2n2w\" (UID: \"66f3af82-eb7e-45c5-b6b5-20616e528b53\") " pod="opendatahub/maas-controller-fcb85f65f-c2n2w" Jul 28 22:18:07.770362 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:07.770341 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d6457fd9-7kwsd"] Jul 28 22:18:07.866859 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:07.866769 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbcn4\" (UniqueName: \"kubernetes.io/projected/88b2f7fd-46ba-4f7f-a566-50056f6fbd79-kube-api-access-tbcn4\") pod \"maas-controller-d6457fd9-7kwsd\" (UID: \"88b2f7fd-46ba-4f7f-a566-50056f6fbd79\") " pod="opendatahub/maas-controller-d6457fd9-7kwsd" Jul 28 22:18:07.866859 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:07.866852 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/66f3af82-eb7e-45c5-b6b5-20616e528b53-webhook-cert\") pod \"maas-controller-fcb85f65f-c2n2w\" (UID: \"66f3af82-eb7e-45c5-b6b5-20616e528b53\") " pod="opendatahub/maas-controller-fcb85f65f-c2n2w" Jul 28 22:18:07.867047 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:07.866873 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46btn\" (UniqueName: \"kubernetes.io/projected/66f3af82-eb7e-45c5-b6b5-20616e528b53-kube-api-access-46btn\") pod \"maas-controller-fcb85f65f-c2n2w\" (UID: \"66f3af82-eb7e-45c5-b6b5-20616e528b53\") " pod="opendatahub/maas-controller-fcb85f65f-c2n2w" Jul 28 22:18:07.867047 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:07.866909 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/88b2f7fd-46ba-4f7f-a566-50056f6fbd79-webhook-cert\") pod \"maas-controller-d6457fd9-7kwsd\" (UID: \"88b2f7fd-46ba-4f7f-a566-50056f6fbd79\") " pod="opendatahub/maas-controller-d6457fd9-7kwsd" Jul 28 22:18:07.867047 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:18:07.867008 2553 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 28 22:18:07.867145 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:18:07.867081 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66f3af82-eb7e-45c5-b6b5-20616e528b53-webhook-cert podName:66f3af82-eb7e-45c5-b6b5-20616e528b53 nodeName:}" failed. No retries permitted until 2026-07-28 22:18:08.367065221 +0000 UTC m=+508.454323268 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/66f3af82-eb7e-45c5-b6b5-20616e528b53-webhook-cert") pod "maas-controller-fcb85f65f-c2n2w" (UID: "66f3af82-eb7e-45c5-b6b5-20616e528b53") : secret "maas-controller-webhook-cert" not found Jul 28 22:18:07.875305 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:07.875277 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46btn\" (UniqueName: \"kubernetes.io/projected/66f3af82-eb7e-45c5-b6b5-20616e528b53-kube-api-access-46btn\") pod \"maas-controller-fcb85f65f-c2n2w\" (UID: \"66f3af82-eb7e-45c5-b6b5-20616e528b53\") " pod="opendatahub/maas-controller-fcb85f65f-c2n2w" Jul 28 22:18:07.967777 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:07.967699 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/88b2f7fd-46ba-4f7f-a566-50056f6fbd79-webhook-cert\") pod \"maas-controller-d6457fd9-7kwsd\" (UID: \"88b2f7fd-46ba-4f7f-a566-50056f6fbd79\") " pod="opendatahub/maas-controller-d6457fd9-7kwsd" Jul 28 22:18:07.967987 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:07.967807 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbcn4\" (UniqueName: \"kubernetes.io/projected/88b2f7fd-46ba-4f7f-a566-50056f6fbd79-kube-api-access-tbcn4\") pod \"maas-controller-d6457fd9-7kwsd\" (UID: \"88b2f7fd-46ba-4f7f-a566-50056f6fbd79\") " pod="opendatahub/maas-controller-d6457fd9-7kwsd" Jul 28 22:18:07.967987 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:18:07.967852 2553 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 28 22:18:07.967987 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:18:07.967949 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88b2f7fd-46ba-4f7f-a566-50056f6fbd79-webhook-cert podName:88b2f7fd-46ba-4f7f-a566-50056f6fbd79 nodeName:}" failed. No retries permitted until 2026-07-28 22:18:08.467925696 +0000 UTC m=+508.555183743 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/88b2f7fd-46ba-4f7f-a566-50056f6fbd79-webhook-cert") pod "maas-controller-d6457fd9-7kwsd" (UID: "88b2f7fd-46ba-4f7f-a566-50056f6fbd79") : secret "maas-controller-webhook-cert" not found Jul 28 22:18:07.976365 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:07.976342 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbcn4\" (UniqueName: \"kubernetes.io/projected/88b2f7fd-46ba-4f7f-a566-50056f6fbd79-kube-api-access-tbcn4\") pod \"maas-controller-d6457fd9-7kwsd\" (UID: \"88b2f7fd-46ba-4f7f-a566-50056f6fbd79\") " pod="opendatahub/maas-controller-d6457fd9-7kwsd" Jul 28 22:18:08.372058 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:08.372021 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/66f3af82-eb7e-45c5-b6b5-20616e528b53-webhook-cert\") pod \"maas-controller-fcb85f65f-c2n2w\" (UID: \"66f3af82-eb7e-45c5-b6b5-20616e528b53\") " pod="opendatahub/maas-controller-fcb85f65f-c2n2w" Jul 28 22:18:08.374533 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:08.374512 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/66f3af82-eb7e-45c5-b6b5-20616e528b53-webhook-cert\") pod \"maas-controller-fcb85f65f-c2n2w\" (UID: \"66f3af82-eb7e-45c5-b6b5-20616e528b53\") " pod="opendatahub/maas-controller-fcb85f65f-c2n2w" Jul 28 22:18:08.472804 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:08.472773 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/88b2f7fd-46ba-4f7f-a566-50056f6fbd79-webhook-cert\") pod \"maas-controller-d6457fd9-7kwsd\" (UID: \"88b2f7fd-46ba-4f7f-a566-50056f6fbd79\") " pod="opendatahub/maas-controller-d6457fd9-7kwsd" Jul 28 22:18:08.475279 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:08.475250 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/88b2f7fd-46ba-4f7f-a566-50056f6fbd79-webhook-cert\") pod \"maas-controller-d6457fd9-7kwsd\" (UID: \"88b2f7fd-46ba-4f7f-a566-50056f6fbd79\") " pod="opendatahub/maas-controller-d6457fd9-7kwsd" Jul 28 22:18:08.499174 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:08.499150 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-fcb85f65f-c2n2w" Jul 28 22:18:08.634060 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:08.633971 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-fcb85f65f-c2n2w"] Jul 28 22:18:08.636260 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:18:08.636238 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66f3af82_eb7e_45c5_b6b5_20616e528b53.slice/crio-e2c07726cd3d6eaf4c1ec869e12b7b4dde5b20de868464ba37199d30e5f5db8e WatchSource:0}: Error finding container e2c07726cd3d6eaf4c1ec869e12b7b4dde5b20de868464ba37199d30e5f5db8e: Status 404 returned error can't find the container with id e2c07726cd3d6eaf4c1ec869e12b7b4dde5b20de868464ba37199d30e5f5db8e Jul 28 22:18:08.670844 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:08.670822 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d6457fd9-7kwsd" Jul 28 22:18:08.801253 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:08.801228 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d6457fd9-7kwsd"] Jul 28 22:18:08.802924 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:18:08.802898 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88b2f7fd_46ba_4f7f_a566_50056f6fbd79.slice/crio-6bcea0d8c5020a0b289b50960ae687d8994ae7793c6c279bea69b91d78c2de9d WatchSource:0}: Error finding container 6bcea0d8c5020a0b289b50960ae687d8994ae7793c6c279bea69b91d78c2de9d: Status 404 returned error can't find the container with id 6bcea0d8c5020a0b289b50960ae687d8994ae7793c6c279bea69b91d78c2de9d Jul 28 22:18:08.895324 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:08.895248 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-fcb85f65f-c2n2w" event={"ID":"66f3af82-eb7e-45c5-b6b5-20616e528b53","Type":"ContainerStarted","Data":"e2c07726cd3d6eaf4c1ec869e12b7b4dde5b20de868464ba37199d30e5f5db8e"} Jul 28 22:18:08.896343 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:08.896317 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d6457fd9-7kwsd" event={"ID":"88b2f7fd-46ba-4f7f-a566-50056f6fbd79","Type":"ContainerStarted","Data":"6bcea0d8c5020a0b289b50960ae687d8994ae7793c6c279bea69b91d78c2de9d"} Jul 28 22:18:11.920792 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:11.920742 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d6457fd9-7kwsd" event={"ID":"88b2f7fd-46ba-4f7f-a566-50056f6fbd79","Type":"ContainerStarted","Data":"e8d6bf258c79eabb40453b2c49428f7a650fd65cde921241d7583b7a06c8bae6"} Jul 28 22:18:11.921255 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:11.920820 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-d6457fd9-7kwsd" Jul 28 22:18:11.923096 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:11.923052 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-fcb85f65f-c2n2w" event={"ID":"66f3af82-eb7e-45c5-b6b5-20616e528b53","Type":"ContainerStarted","Data":"907c7431ce0d8667e8c1b6b1976c55e7e1721469c395ef7c87d93d390200814c"} Jul 28 22:18:11.923249 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:11.923164 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-fcb85f65f-c2n2w" Jul 28 22:18:11.942626 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:11.942577 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-d6457fd9-7kwsd" podStartSLOduration=2.453911685 podStartE2EDuration="4.942560343s" totalImagesPullingTime="2.488648658s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:18:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:18:11.941119276 +0000 UTC m=+512.028377346" watchObservedRunningTime="2026-07-28 22:18:11.942560343 +0000 UTC m=+512.029818422" Jul 28 22:18:11.958563 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:11.958510 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-fcb85f65f-c2n2w" podStartSLOduration=2.296035629 podStartE2EDuration="4.958493552s" totalImagesPullingTime="2.662457923s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:18:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:18:11.957890315 +0000 UTC m=+512.045148386" watchObservedRunningTime="2026-07-28 22:18:11.958493552 +0000 UTC m=+512.045751620" Jul 28 22:18:22.932224 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:22.932192 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-fcb85f65f-c2n2w" Jul 28 22:18:22.932651 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:22.932262 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-d6457fd9-7kwsd" Jul 28 22:18:22.986858 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:22.986818 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-fcb85f65f-c2n2w"] Jul 28 22:18:22.987177 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:22.987138 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-fcb85f65f-c2n2w" podUID="66f3af82-eb7e-45c5-b6b5-20616e528b53" containerName="manager" containerID="cri-o://907c7431ce0d8667e8c1b6b1976c55e7e1721469c395ef7c87d93d390200814c" gracePeriod=30 Jul 28 22:18:23.231586 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:23.231561 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-fcb85f65f-c2n2w" Jul 28 22:18:23.300244 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:23.300208 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/66f3af82-eb7e-45c5-b6b5-20616e528b53-webhook-cert\") pod \"66f3af82-eb7e-45c5-b6b5-20616e528b53\" (UID: \"66f3af82-eb7e-45c5-b6b5-20616e528b53\") " Jul 28 22:18:23.300413 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:23.300286 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46btn\" (UniqueName: \"kubernetes.io/projected/66f3af82-eb7e-45c5-b6b5-20616e528b53-kube-api-access-46btn\") pod \"66f3af82-eb7e-45c5-b6b5-20616e528b53\" (UID: \"66f3af82-eb7e-45c5-b6b5-20616e528b53\") " Jul 28 22:18:23.302623 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:23.302591 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66f3af82-eb7e-45c5-b6b5-20616e528b53-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "66f3af82-eb7e-45c5-b6b5-20616e528b53" (UID: "66f3af82-eb7e-45c5-b6b5-20616e528b53"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 22:18:23.302759 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:23.302731 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66f3af82-eb7e-45c5-b6b5-20616e528b53-kube-api-access-46btn" (OuterVolumeSpecName: "kube-api-access-46btn") pod "66f3af82-eb7e-45c5-b6b5-20616e528b53" (UID: "66f3af82-eb7e-45c5-b6b5-20616e528b53"). InnerVolumeSpecName "kube-api-access-46btn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:18:23.401783 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:23.401751 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46btn\" (UniqueName: \"kubernetes.io/projected/66f3af82-eb7e-45c5-b6b5-20616e528b53-kube-api-access-46btn\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:18:23.401783 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:23.401781 2553 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/66f3af82-eb7e-45c5-b6b5-20616e528b53-webhook-cert\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:18:23.987101 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:23.987070 2553 generic.go:356] "Generic (PLEG): container finished" podID="66f3af82-eb7e-45c5-b6b5-20616e528b53" containerID="907c7431ce0d8667e8c1b6b1976c55e7e1721469c395ef7c87d93d390200814c" exitCode=0 Jul 28 22:18:23.987532 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:23.987114 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-fcb85f65f-c2n2w" Jul 28 22:18:23.987532 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:23.987154 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-fcb85f65f-c2n2w" event={"ID":"66f3af82-eb7e-45c5-b6b5-20616e528b53","Type":"ContainerDied","Data":"907c7431ce0d8667e8c1b6b1976c55e7e1721469c395ef7c87d93d390200814c"} Jul 28 22:18:23.987532 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:23.987213 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-fcb85f65f-c2n2w" event={"ID":"66f3af82-eb7e-45c5-b6b5-20616e528b53","Type":"ContainerDied","Data":"e2c07726cd3d6eaf4c1ec869e12b7b4dde5b20de868464ba37199d30e5f5db8e"} Jul 28 22:18:23.987532 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:23.987235 2553 scope.go:117] "RemoveContainer" containerID="907c7431ce0d8667e8c1b6b1976c55e7e1721469c395ef7c87d93d390200814c" Jul 28 22:18:23.996356 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:23.996340 2553 scope.go:117] "RemoveContainer" containerID="907c7431ce0d8667e8c1b6b1976c55e7e1721469c395ef7c87d93d390200814c" Jul 28 22:18:23.996615 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:18:23.996600 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"907c7431ce0d8667e8c1b6b1976c55e7e1721469c395ef7c87d93d390200814c\": container with ID starting with 907c7431ce0d8667e8c1b6b1976c55e7e1721469c395ef7c87d93d390200814c not found: ID does not exist" containerID="907c7431ce0d8667e8c1b6b1976c55e7e1721469c395ef7c87d93d390200814c" Jul 28 22:18:23.996671 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:23.996625 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"907c7431ce0d8667e8c1b6b1976c55e7e1721469c395ef7c87d93d390200814c"} err="failed to get container status \"907c7431ce0d8667e8c1b6b1976c55e7e1721469c395ef7c87d93d390200814c\": rpc error: code = NotFound desc = could not find container \"907c7431ce0d8667e8c1b6b1976c55e7e1721469c395ef7c87d93d390200814c\": container with ID starting with 907c7431ce0d8667e8c1b6b1976c55e7e1721469c395ef7c87d93d390200814c not found: ID does not exist" Jul 28 22:18:24.013351 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:24.013329 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-fcb85f65f-c2n2w"] Jul 28 22:18:24.015228 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:24.015209 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-fcb85f65f-c2n2w"] Jul 28 22:18:24.404650 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:24.404571 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66f3af82-eb7e-45c5-b6b5-20616e528b53" path="/var/lib/kubelet/pods/66f3af82-eb7e-45c5-b6b5-20616e528b53/volumes" Jul 28 22:18:43.770641 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:43.770601 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5f4497d45c-qdp9t"] Jul 28 22:18:43.771207 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:43.770990 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66f3af82-eb7e-45c5-b6b5-20616e528b53" containerName="manager" Jul 28 22:18:43.771207 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:43.771006 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="66f3af82-eb7e-45c5-b6b5-20616e528b53" containerName="manager" Jul 28 22:18:43.771207 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:43.771080 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="66f3af82-eb7e-45c5-b6b5-20616e528b53" containerName="manager" Jul 28 22:18:43.775256 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:43.775237 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5f4497d45c-qdp9t" Jul 28 22:18:43.777518 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:43.777490 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 28 22:18:43.777663 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:43.777538 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 22:18:43.777663 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:43.777563 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-89lsd\"" Jul 28 22:18:43.777663 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:43.777616 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 28 22:18:43.786863 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:43.786840 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5f4497d45c-qdp9t"] Jul 28 22:18:43.867190 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:43.867139 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49752556-bd0b-46ea-b3fd-d4b807db64c7-maas-api-tls\") pod \"maas-api-5f4497d45c-qdp9t\" (UID: \"49752556-bd0b-46ea-b3fd-d4b807db64c7\") " pod="odh-ai-gateway-infra/maas-api-5f4497d45c-qdp9t" Jul 28 22:18:43.867375 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:43.867271 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkrjd\" (UniqueName: \"kubernetes.io/projected/49752556-bd0b-46ea-b3fd-d4b807db64c7-kube-api-access-fkrjd\") pod \"maas-api-5f4497d45c-qdp9t\" (UID: \"49752556-bd0b-46ea-b3fd-d4b807db64c7\") " pod="odh-ai-gateway-infra/maas-api-5f4497d45c-qdp9t" Jul 28 22:18:43.867375 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:43.867309 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49752556-bd0b-46ea-b3fd-d4b807db64c7-openshift-service-ca\") pod \"maas-api-5f4497d45c-qdp9t\" (UID: \"49752556-bd0b-46ea-b3fd-d4b807db64c7\") " pod="odh-ai-gateway-infra/maas-api-5f4497d45c-qdp9t" Jul 28 22:18:43.968254 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:43.968215 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkrjd\" (UniqueName: \"kubernetes.io/projected/49752556-bd0b-46ea-b3fd-d4b807db64c7-kube-api-access-fkrjd\") pod \"maas-api-5f4497d45c-qdp9t\" (UID: \"49752556-bd0b-46ea-b3fd-d4b807db64c7\") " pod="odh-ai-gateway-infra/maas-api-5f4497d45c-qdp9t" Jul 28 22:18:43.968456 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:43.968266 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49752556-bd0b-46ea-b3fd-d4b807db64c7-openshift-service-ca\") pod \"maas-api-5f4497d45c-qdp9t\" (UID: \"49752556-bd0b-46ea-b3fd-d4b807db64c7\") " pod="odh-ai-gateway-infra/maas-api-5f4497d45c-qdp9t" Jul 28 22:18:43.968456 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:43.968291 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49752556-bd0b-46ea-b3fd-d4b807db64c7-maas-api-tls\") pod \"maas-api-5f4497d45c-qdp9t\" (UID: \"49752556-bd0b-46ea-b3fd-d4b807db64c7\") " pod="odh-ai-gateway-infra/maas-api-5f4497d45c-qdp9t" Jul 28 22:18:43.968456 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:18:43.968400 2553 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found Jul 28 22:18:43.968456 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:18:43.968451 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49752556-bd0b-46ea-b3fd-d4b807db64c7-maas-api-tls podName:49752556-bd0b-46ea-b3fd-d4b807db64c7 nodeName:}" failed. No retries permitted until 2026-07-28 22:18:44.468436149 +0000 UTC m=+544.555694200 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/49752556-bd0b-46ea-b3fd-d4b807db64c7-maas-api-tls") pod "maas-api-5f4497d45c-qdp9t" (UID: "49752556-bd0b-46ea-b3fd-d4b807db64c7") : secret "maas-api-serving-cert" not found Jul 28 22:18:43.969001 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:43.968976 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49752556-bd0b-46ea-b3fd-d4b807db64c7-openshift-service-ca\") pod \"maas-api-5f4497d45c-qdp9t\" (UID: \"49752556-bd0b-46ea-b3fd-d4b807db64c7\") " pod="odh-ai-gateway-infra/maas-api-5f4497d45c-qdp9t" Jul 28 22:18:43.979423 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:43.979398 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkrjd\" (UniqueName: \"kubernetes.io/projected/49752556-bd0b-46ea-b3fd-d4b807db64c7-kube-api-access-fkrjd\") pod \"maas-api-5f4497d45c-qdp9t\" (UID: \"49752556-bd0b-46ea-b3fd-d4b807db64c7\") " pod="odh-ai-gateway-infra/maas-api-5f4497d45c-qdp9t" Jul 28 22:18:44.473421 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:44.473382 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49752556-bd0b-46ea-b3fd-d4b807db64c7-maas-api-tls\") pod \"maas-api-5f4497d45c-qdp9t\" (UID: \"49752556-bd0b-46ea-b3fd-d4b807db64c7\") " pod="odh-ai-gateway-infra/maas-api-5f4497d45c-qdp9t" Jul 28 22:18:44.476075 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:44.476049 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49752556-bd0b-46ea-b3fd-d4b807db64c7-maas-api-tls\") pod \"maas-api-5f4497d45c-qdp9t\" (UID: \"49752556-bd0b-46ea-b3fd-d4b807db64c7\") " pod="odh-ai-gateway-infra/maas-api-5f4497d45c-qdp9t" Jul 28 22:18:44.686081 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:44.686048 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5f4497d45c-qdp9t" Jul 28 22:18:44.826868 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:44.826650 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5f4497d45c-qdp9t"] Jul 28 22:18:44.829887 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:18:44.829859 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49752556_bd0b_46ea_b3fd_d4b807db64c7.slice/crio-e86abe50e32d7b1b7f27b234b217c37d2a915b65075ddf590d89c3e6aedf537f WatchSource:0}: Error finding container e86abe50e32d7b1b7f27b234b217c37d2a915b65075ddf590d89c3e6aedf537f: Status 404 returned error can't find the container with id e86abe50e32d7b1b7f27b234b217c37d2a915b65075ddf590d89c3e6aedf537f Jul 28 22:18:45.096147 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:45.096063 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5f4497d45c-qdp9t" event={"ID":"49752556-bd0b-46ea-b3fd-d4b807db64c7","Type":"ContainerStarted","Data":"e86abe50e32d7b1b7f27b234b217c37d2a915b65075ddf590d89c3e6aedf537f"} Jul 28 22:18:47.108383 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:47.108345 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5f4497d45c-qdp9t" event={"ID":"49752556-bd0b-46ea-b3fd-d4b807db64c7","Type":"ContainerStarted","Data":"722dd188e2000bacce2e048ae9cb4eb7c458b14ba621d4fd31fdf9083ad2c421"} Jul 28 22:18:47.108878 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:47.108434 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5f4497d45c-qdp9t" Jul 28 22:18:47.128103 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:47.128038 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5f4497d45c-qdp9t" podStartSLOduration=2.45832175 podStartE2EDuration="4.128015519s" totalImagesPullingTime="1.669693769s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:18:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:18:47.124920753 +0000 UTC m=+547.212178825" watchObservedRunningTime="2026-07-28 22:18:47.128015519 +0000 UTC m=+547.215273591" Jul 28 22:18:52.136220 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:52.136127 2553 generic.go:356] "Generic (PLEG): container finished" podID="e776e265-c83c-42be-be8c-07647513e2ba" containerID="2d7f3efd95e65b27dccd3b54ac09a9b5deca8c3218bbddce1393dd47442229df" exitCode=0 Jul 28 22:18:52.136220 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:52.136206 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfrsf" event={"ID":"e776e265-c83c-42be-be8c-07647513e2ba","Type":"ContainerDied","Data":"2d7f3efd95e65b27dccd3b54ac09a9b5deca8c3218bbddce1393dd47442229df"} Jul 28 22:18:52.136645 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:52.136596 2553 scope.go:117] "RemoveContainer" containerID="2d7f3efd95e65b27dccd3b54ac09a9b5deca8c3218bbddce1393dd47442229df" Jul 28 22:18:52.137540 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:52.137514 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:18:52.137639 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:52.137582 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:18:52.137639 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:52.137611 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:18:53.129944 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:53.129913 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5f4497d45c-qdp9t" Jul 28 22:18:53.144051 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:53.144016 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfrsf" event={"ID":"e776e265-c83c-42be-be8c-07647513e2ba","Type":"ContainerStarted","Data":"47268b18b828b238393d360e02d563dc637ec05b34cfe924e7ece6310349f645"} Jul 28 22:18:53.240956 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:53.240918 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfrsf" Jul 28 22:18:53.242272 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:53.242246 2553 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-bfrsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Jul 28 22:18:53.242391 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:53.242300 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfrsf" podUID="e776e265-c83c-42be-be8c-07647513e2ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Jul 28 22:18:54.240884 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:54.240847 2553 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-bfrsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Jul 28 22:18:54.241274 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:54.240919 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfrsf" podUID="e776e265-c83c-42be-be8c-07647513e2ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Jul 28 22:18:55.241404 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:55.241369 2553 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-bfrsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Jul 28 22:18:55.241832 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:55.241438 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfrsf" podUID="e776e265-c83c-42be-be8c-07647513e2ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Jul 28 22:18:56.240860 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:56.240827 2553 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-bfrsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Jul 28 22:18:56.241012 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:56.240882 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfrsf" podUID="e776e265-c83c-42be-be8c-07647513e2ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Jul 28 22:18:57.245017 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:57.244988 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfrsf" Jul 28 22:18:57.245422 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:57.245270 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfrsf" Jul 28 22:18:57.245983 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:18:57.245966 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bfrsf" Jul 28 22:19:03.685127 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:19:03.685089 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7rsjb"] Jul 28 22:19:03.689191 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:19:03.689169 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7rsjb" Jul 28 22:19:03.692113 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:19:03.692088 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 28 22:19:03.692242 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:19:03.692109 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-72d9n\"" Jul 28 22:19:03.692242 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:19:03.692109 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 28 22:19:03.692242 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:19:03.692088 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 28 22:19:03.698653 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:19:03.698243 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7rsjb"] Jul 28 22:19:03.844277 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:19:03.844241 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75574aba-fe8f-451e-95f1-d11be8d7249a-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7rsjb\" (UID: \"75574aba-fe8f-451e-95f1-d11be8d7249a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7rsjb" Jul 28 22:19:03.844277 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:19:03.844284 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75574aba-fe8f-451e-95f1-d11be8d7249a-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7rsjb\" (UID: \"75574aba-fe8f-451e-95f1-d11be8d7249a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7rsjb" Jul 28 22:19:03.844536 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:19:03.844309 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75574aba-fe8f-451e-95f1-d11be8d7249a-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7rsjb\" (UID: \"75574aba-fe8f-451e-95f1-d11be8d7249a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7rsjb" Jul 28 22:19:03.844536 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:19:03.844391 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75574aba-fe8f-451e-95f1-d11be8d7249a-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7rsjb\" (UID: \"75574aba-fe8f-451e-95f1-d11be8d7249a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7rsjb" Jul 28 22:19:03.844536 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:19:03.844434 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb5bs\" (UniqueName: \"kubernetes.io/projected/75574aba-fe8f-451e-95f1-d11be8d7249a-kube-api-access-xb5bs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7rsjb\" (UID: \"75574aba-fe8f-451e-95f1-d11be8d7249a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7rsjb" Jul 28 22:19:03.844536 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:19:03.844479 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75574aba-fe8f-451e-95f1-d11be8d7249a-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7rsjb\" (UID: \"75574aba-fe8f-451e-95f1-d11be8d7249a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7rsjb" Jul 28 22:19:03.945665 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:19:03.945573 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75574aba-fe8f-451e-95f1-d11be8d7249a-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7rsjb\" (UID: \"75574aba-fe8f-451e-95f1-d11be8d7249a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7rsjb" Jul 28 22:19:03.945665 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:19:03.945622 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75574aba-fe8f-451e-95f1-d11be8d7249a-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7rsjb\" (UID: \"75574aba-fe8f-451e-95f1-d11be8d7249a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7rsjb" Jul 28 22:19:03.945665 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:19:03.945648 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb5bs\" (UniqueName: \"kubernetes.io/projected/75574aba-fe8f-451e-95f1-d11be8d7249a-kube-api-access-xb5bs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7rsjb\" (UID: \"75574aba-fe8f-451e-95f1-d11be8d7249a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7rsjb" Jul 28 22:19:03.945985 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:19:03.945688 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75574aba-fe8f-451e-95f1-d11be8d7249a-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7rsjb\" (UID: \"75574aba-fe8f-451e-95f1-d11be8d7249a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7rsjb" Jul 28 22:19:03.945985 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:19:03.945791 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75574aba-fe8f-451e-95f1-d11be8d7249a-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7rsjb\" (UID: \"75574aba-fe8f-451e-95f1-d11be8d7249a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7rsjb" Jul 28 22:19:03.945985 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:19:03.945835 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75574aba-fe8f-451e-95f1-d11be8d7249a-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7rsjb\" (UID: \"75574aba-fe8f-451e-95f1-d11be8d7249a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7rsjb" Jul 28 22:19:03.946150 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:19:03.946082 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75574aba-fe8f-451e-95f1-d11be8d7249a-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7rsjb\" (UID: \"75574aba-fe8f-451e-95f1-d11be8d7249a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7rsjb" Jul 28 22:19:03.946150 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:19:03.946143 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75574aba-fe8f-451e-95f1-d11be8d7249a-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7rsjb\" (UID: \"75574aba-fe8f-451e-95f1-d11be8d7249a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7rsjb" Jul 28 22:19:03.946228 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:19:03.946183 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75574aba-fe8f-451e-95f1-d11be8d7249a-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7rsjb\" (UID: \"75574aba-fe8f-451e-95f1-d11be8d7249a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7rsjb" Jul 28 22:19:03.948030 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:19:03.948007 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75574aba-fe8f-451e-95f1-d11be8d7249a-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7rsjb\" (UID: \"75574aba-fe8f-451e-95f1-d11be8d7249a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7rsjb" Jul 28 22:19:03.948284 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:19:03.948268 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75574aba-fe8f-451e-95f1-d11be8d7249a-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7rsjb\" (UID: \"75574aba-fe8f-451e-95f1-d11be8d7249a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7rsjb" Jul 28 22:19:03.953764 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:19:03.953743 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb5bs\" (UniqueName: \"kubernetes.io/projected/75574aba-fe8f-451e-95f1-d11be8d7249a-kube-api-access-xb5bs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7rsjb\" (UID: \"75574aba-fe8f-451e-95f1-d11be8d7249a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7rsjb" Jul 28 22:19:04.002480 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:19:04.002450 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7rsjb" Jul 28 22:19:04.144099 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:19:04.144072 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7rsjb"] Jul 28 22:19:04.145838 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:19:04.145810 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75574aba_fe8f_451e_95f1_d11be8d7249a.slice/crio-d3125a76a9ece5548cd251cbaccbe2f5ce8a670423a8920542027d1d44930af0 WatchSource:0}: Error finding container d3125a76a9ece5548cd251cbaccbe2f5ce8a670423a8920542027d1d44930af0: Status 404 returned error can't find the container with id d3125a76a9ece5548cd251cbaccbe2f5ce8a670423a8920542027d1d44930af0 Jul 28 22:19:04.200418 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:19:04.200334 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7rsjb" event={"ID":"75574aba-fe8f-451e-95f1-d11be8d7249a","Type":"ContainerStarted","Data":"d3125a76a9ece5548cd251cbaccbe2f5ce8a670423a8920542027d1d44930af0"} Jul 28 22:19:06.213760 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:19:06.213697 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7rsjb" event={"ID":"75574aba-fe8f-451e-95f1-d11be8d7249a","Type":"ContainerStarted","Data":"aad99dd9f88b40430d3036886269a73ebe4dd3350c9ed44f902b5cc031b64a3f"} Jul 28 22:19:06.229404 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:19:06.229356 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7rsjb" podStartSLOduration=2.189979212 podStartE2EDuration="3.229339856s" totalImagesPullingTime="1.039360644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:19:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:19:06.228522667 +0000 UTC m=+566.315780737" watchObservedRunningTime="2026-07-28 22:19:06.229339856 +0000 UTC m=+566.316597924" Jul 28 22:19:14.002981 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:19:14.002944 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7rsjb" Jul 28 22:19:14.003511 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:19:14.003023 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7rsjb" Jul 28 22:19:14.015772 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:19:14.015748 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7rsjb" Jul 28 22:19:14.270250 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:19:14.270176 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7rsjb" Jul 28 22:19:32.664487 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:19:32.664451 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbwgdp"] Jul 28 22:19:32.667896 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:19:32.667876 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbwgdp" Jul 28 22:19:32.670202 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:19:32.670186 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 28 22:19:32.679376 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:19:32.679354 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbwgdp"] Jul 28 22:19:32.797315 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:19:32.797285 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7adda96-3dfd-42f9-987e-2aeed5d39c49-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbwgdp\" (UID: \"d7adda96-3dfd-42f9-987e-2aeed5d39c49\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbwgdp" Jul 28 22:19:32.797315 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:19:32.797319 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7adda96-3dfd-42f9-987e-2aeed5d39c49-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbwgdp\" (UID: \"d7adda96-3dfd-42f9-987e-2aeed5d39c49\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbwgdp" Jul 28 22:19:32.797538 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:19:32.797345 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7adda96-3dfd-42f9-987e-2aeed5d39c49-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbwgdp\" (UID: \"d7adda96-3dfd-42f9-987e-2aeed5d39c49\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbwgdp" Jul 28 22:19:32.797538 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:19:32.797429 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7adda96-3dfd-42f9-987e-2aeed5d39c49-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbwgdp\" (UID: \"d7adda96-3dfd-42f9-987e-2aeed5d39c49\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbwgdp" Jul 28 22:19:32.797538 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:19:32.797474 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7adda96-3dfd-42f9-987e-2aeed5d39c49-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbwgdp\" (UID: \"d7adda96-3dfd-42f9-987e-2aeed5d39c49\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbwgdp" Jul 28 22:19:32.797538 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:19:32.797495 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd65m\" (UniqueName: \"kubernetes.io/projected/d7adda96-3dfd-42f9-987e-2aeed5d39c49-kube-api-access-cd65m\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbwgdp\" (UID: \"d7adda96-3dfd-42f9-987e-2aeed5d39c49\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbwgdp" Jul 28 22:19:32.898570 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:19:32.898529 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7adda96-3dfd-42f9-987e-2aeed5d39c49-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbwgdp\" (UID: \"d7adda96-3dfd-42f9-987e-2aeed5d39c49\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbwgdp" Jul 28 22:19:32.898791 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:19:32.898596 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7adda96-3dfd-42f9-987e-2aeed5d39c49-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbwgdp\" (UID: \"d7adda96-3dfd-42f9-987e-2aeed5d39c49\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbwgdp" Jul 28 22:19:32.898791 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:19:32.898614 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd65m\" (UniqueName: \"kubernetes.io/projected/d7adda96-3dfd-42f9-987e-2aeed5d39c49-kube-api-access-cd65m\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbwgdp\" (UID: \"d7adda96-3dfd-42f9-987e-2aeed5d39c49\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbwgdp" Jul 28 22:19:32.898791 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:19:32.898641 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7adda96-3dfd-42f9-987e-2aeed5d39c49-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbwgdp\" (UID: \"d7adda96-3dfd-42f9-987e-2aeed5d39c49\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbwgdp" Jul 28 22:19:32.898791 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:19:32.898663 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7adda96-3dfd-42f9-987e-2aeed5d39c49-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbwgdp\" (UID: \"d7adda96-3dfd-42f9-987e-2aeed5d39c49\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbwgdp" Jul 28 22:19:32.898791 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:19:32.898731 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7adda96-3dfd-42f9-987e-2aeed5d39c49-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbwgdp\" (UID: \"d7adda96-3dfd-42f9-987e-2aeed5d39c49\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbwgdp" Jul 28 22:19:32.899143 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:19:32.899119 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7adda96-3dfd-42f9-987e-2aeed5d39c49-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbwgdp\" (UID: \"d7adda96-3dfd-42f9-987e-2aeed5d39c49\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbwgdp" Jul 28 22:19:32.899208 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:19:32.899180 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7adda96-3dfd-42f9-987e-2aeed5d39c49-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbwgdp\" (UID: \"d7adda96-3dfd-42f9-987e-2aeed5d39c49\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbwgdp" Jul 28 22:19:32.899246 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:19:32.899208 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7adda96-3dfd-42f9-987e-2aeed5d39c49-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbwgdp\" (UID: \"d7adda96-3dfd-42f9-987e-2aeed5d39c49\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbwgdp" Jul 28 22:19:32.900998 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:19:32.900980 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7adda96-3dfd-42f9-987e-2aeed5d39c49-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbwgdp\" (UID: \"d7adda96-3dfd-42f9-987e-2aeed5d39c49\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbwgdp" Jul 28 22:19:32.901131 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:19:32.901115 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7adda96-3dfd-42f9-987e-2aeed5d39c49-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbwgdp\" (UID: \"d7adda96-3dfd-42f9-987e-2aeed5d39c49\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbwgdp" Jul 28 22:19:32.906917 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:19:32.906898 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd65m\" (UniqueName: \"kubernetes.io/projected/d7adda96-3dfd-42f9-987e-2aeed5d39c49-kube-api-access-cd65m\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbwgdp\" (UID: \"d7adda96-3dfd-42f9-987e-2aeed5d39c49\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbwgdp" Jul 28 22:19:32.979556 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:19:32.979521 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbwgdp" Jul 28 22:19:33.116802 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:19:33.116763 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbwgdp"] Jul 28 22:19:33.118505 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:19:33.118469 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7adda96_3dfd_42f9_987e_2aeed5d39c49.slice/crio-6be745b8959d8a075151e78ab82446d1e740a8785a3f51f42ef1b7907c72831e WatchSource:0}: Error finding container 6be745b8959d8a075151e78ab82446d1e740a8785a3f51f42ef1b7907c72831e: Status 404 returned error can't find the container with id 6be745b8959d8a075151e78ab82446d1e740a8785a3f51f42ef1b7907c72831e Jul 28 22:19:33.355551 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:19:33.355510 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbwgdp" event={"ID":"d7adda96-3dfd-42f9-987e-2aeed5d39c49","Type":"ContainerStarted","Data":"6be745b8959d8a075151e78ab82446d1e740a8785a3f51f42ef1b7907c72831e"} Jul 28 22:19:34.361988 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:19:34.361950 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbwgdp" event={"ID":"d7adda96-3dfd-42f9-987e-2aeed5d39c49","Type":"ContainerStarted","Data":"893c12d95a7f4e4d7a43dcafea1f09cda11463b5fdce3106661ed2373ae3d390"} Jul 28 22:19:40.316859 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:19:40.316832 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mldhp_e1ced996-f136-4d3e-9c85-a5b7809b4517/console-operator/1.log" Jul 28 22:19:40.317298 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:19:40.317222 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mldhp_e1ced996-f136-4d3e-9c85-a5b7809b4517/console-operator/1.log" Jul 28 22:19:42.979845 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:19:42.979811 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbwgdp" Jul 28 22:19:42.980330 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:19:42.979879 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbwgdp" Jul 28 22:19:42.992349 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:19:42.992320 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbwgdp" Jul 28 22:19:43.009423 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:19:43.009377 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbwgdp" podStartSLOduration=10.820893574 podStartE2EDuration="11.009363838s" totalImagesPullingTime="188.470264ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:19:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:19:34.378623658 +0000 UTC m=+594.465881728" watchObservedRunningTime="2026-07-28 22:19:43.009363838 +0000 UTC m=+603.096621907" Jul 28 22:19:43.421061 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:19:43.420975 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbwgdp" Jul 28 22:21:46.598489 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:21:46.598400 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-d6457fd9-7kwsd"] Jul 28 22:21:46.599037 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:21:46.598625 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-d6457fd9-7kwsd" podUID="88b2f7fd-46ba-4f7f-a566-50056f6fbd79" containerName="manager" containerID="cri-o://e8d6bf258c79eabb40453b2c49428f7a650fd65cde921241d7583b7a06c8bae6" gracePeriod=30 Jul 28 22:21:46.855274 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:21:46.855212 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d6457fd9-7kwsd" Jul 28 22:21:46.959783 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:21:46.959744 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tbcn4\" (UniqueName: \"kubernetes.io/projected/88b2f7fd-46ba-4f7f-a566-50056f6fbd79-kube-api-access-tbcn4\") pod \"88b2f7fd-46ba-4f7f-a566-50056f6fbd79\" (UID: \"88b2f7fd-46ba-4f7f-a566-50056f6fbd79\") " Jul 28 22:21:46.959963 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:21:46.959819 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/88b2f7fd-46ba-4f7f-a566-50056f6fbd79-webhook-cert\") pod \"88b2f7fd-46ba-4f7f-a566-50056f6fbd79\" (UID: \"88b2f7fd-46ba-4f7f-a566-50056f6fbd79\") " Jul 28 22:21:46.961921 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:21:46.961895 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88b2f7fd-46ba-4f7f-a566-50056f6fbd79-kube-api-access-tbcn4" (OuterVolumeSpecName: "kube-api-access-tbcn4") pod "88b2f7fd-46ba-4f7f-a566-50056f6fbd79" (UID: "88b2f7fd-46ba-4f7f-a566-50056f6fbd79"). InnerVolumeSpecName "kube-api-access-tbcn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:21:46.962057 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:21:46.961988 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88b2f7fd-46ba-4f7f-a566-50056f6fbd79-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "88b2f7fd-46ba-4f7f-a566-50056f6fbd79" (UID: "88b2f7fd-46ba-4f7f-a566-50056f6fbd79"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 22:21:47.033320 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:21:47.033292 2553 generic.go:356] "Generic (PLEG): container finished" podID="88b2f7fd-46ba-4f7f-a566-50056f6fbd79" containerID="e8d6bf258c79eabb40453b2c49428f7a650fd65cde921241d7583b7a06c8bae6" exitCode=0 Jul 28 22:21:47.033497 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:21:47.033331 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d6457fd9-7kwsd" Jul 28 22:21:47.033497 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:21:47.033378 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d6457fd9-7kwsd" event={"ID":"88b2f7fd-46ba-4f7f-a566-50056f6fbd79","Type":"ContainerDied","Data":"e8d6bf258c79eabb40453b2c49428f7a650fd65cde921241d7583b7a06c8bae6"} Jul 28 22:21:47.033497 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:21:47.033437 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d6457fd9-7kwsd" event={"ID":"88b2f7fd-46ba-4f7f-a566-50056f6fbd79","Type":"ContainerDied","Data":"6bcea0d8c5020a0b289b50960ae687d8994ae7793c6c279bea69b91d78c2de9d"} Jul 28 22:21:47.033497 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:21:47.033464 2553 scope.go:117] "RemoveContainer" containerID="e8d6bf258c79eabb40453b2c49428f7a650fd65cde921241d7583b7a06c8bae6" Jul 28 22:21:47.043253 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:21:47.043233 2553 scope.go:117] "RemoveContainer" containerID="e8d6bf258c79eabb40453b2c49428f7a650fd65cde921241d7583b7a06c8bae6" Jul 28 22:21:47.043526 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:21:47.043508 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8d6bf258c79eabb40453b2c49428f7a650fd65cde921241d7583b7a06c8bae6\": container with ID starting with e8d6bf258c79eabb40453b2c49428f7a650fd65cde921241d7583b7a06c8bae6 not found: ID does not exist" containerID="e8d6bf258c79eabb40453b2c49428f7a650fd65cde921241d7583b7a06c8bae6" Jul 28 22:21:47.043571 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:21:47.043535 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8d6bf258c79eabb40453b2c49428f7a650fd65cde921241d7583b7a06c8bae6"} err="failed to get container status \"e8d6bf258c79eabb40453b2c49428f7a650fd65cde921241d7583b7a06c8bae6\": rpc error: code = NotFound desc = could not find container \"e8d6bf258c79eabb40453b2c49428f7a650fd65cde921241d7583b7a06c8bae6\": container with ID starting with e8d6bf258c79eabb40453b2c49428f7a650fd65cde921241d7583b7a06c8bae6 not found: ID does not exist" Jul 28 22:21:47.057319 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:21:47.057291 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-d6457fd9-7kwsd"] Jul 28 22:21:47.060902 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:21:47.060880 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-d6457fd9-7kwsd"] Jul 28 22:21:47.061048 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:21:47.061029 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbcn4\" (UniqueName: \"kubernetes.io/projected/88b2f7fd-46ba-4f7f-a566-50056f6fbd79-kube-api-access-tbcn4\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:21:47.061095 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:21:47.061053 2553 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/88b2f7fd-46ba-4f7f-a566-50056f6fbd79-webhook-cert\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:21:48.404993 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:21:48.404960 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88b2f7fd-46ba-4f7f-a566-50056f6fbd79" path="/var/lib/kubelet/pods/88b2f7fd-46ba-4f7f-a566-50056f6fbd79/volumes" Jul 28 22:22:47.857460 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:22:47.857423 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-d6457fd9-bvn6q"] Jul 28 22:22:47.857889 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:22:47.857790 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88b2f7fd-46ba-4f7f-a566-50056f6fbd79" containerName="manager" Jul 28 22:22:47.857889 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:22:47.857803 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="88b2f7fd-46ba-4f7f-a566-50056f6fbd79" containerName="manager" Jul 28 22:22:47.857889 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:22:47.857861 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="88b2f7fd-46ba-4f7f-a566-50056f6fbd79" containerName="manager" Jul 28 22:22:47.862076 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:22:47.862061 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d6457fd9-bvn6q" Jul 28 22:22:47.864114 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:22:47.864091 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 28 22:22:47.864217 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:22:47.864094 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 28 22:22:47.864742 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:22:47.864725 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-7k5cv\"" Jul 28 22:22:47.868622 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:22:47.868596 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d6457fd9-bvn6q"] Jul 28 22:22:47.980452 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:22:47.980420 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e298568e-b131-4a29-b479-d7bc20feed12-webhook-cert\") pod \"maas-controller-d6457fd9-bvn6q\" (UID: \"e298568e-b131-4a29-b479-d7bc20feed12\") " pod="opendatahub/maas-controller-d6457fd9-bvn6q" Jul 28 22:22:47.980638 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:22:47.980495 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv46g\" (UniqueName: \"kubernetes.io/projected/e298568e-b131-4a29-b479-d7bc20feed12-kube-api-access-wv46g\") pod \"maas-controller-d6457fd9-bvn6q\" (UID: \"e298568e-b131-4a29-b479-d7bc20feed12\") " pod="opendatahub/maas-controller-d6457fd9-bvn6q" Jul 28 22:22:48.081730 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:22:48.081674 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv46g\" (UniqueName: \"kubernetes.io/projected/e298568e-b131-4a29-b479-d7bc20feed12-kube-api-access-wv46g\") pod \"maas-controller-d6457fd9-bvn6q\" (UID: \"e298568e-b131-4a29-b479-d7bc20feed12\") " pod="opendatahub/maas-controller-d6457fd9-bvn6q" Jul 28 22:22:48.081915 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:22:48.081786 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e298568e-b131-4a29-b479-d7bc20feed12-webhook-cert\") pod \"maas-controller-d6457fd9-bvn6q\" (UID: \"e298568e-b131-4a29-b479-d7bc20feed12\") " pod="opendatahub/maas-controller-d6457fd9-bvn6q" Jul 28 22:22:48.084403 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:22:48.084380 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e298568e-b131-4a29-b479-d7bc20feed12-webhook-cert\") pod \"maas-controller-d6457fd9-bvn6q\" (UID: \"e298568e-b131-4a29-b479-d7bc20feed12\") " pod="opendatahub/maas-controller-d6457fd9-bvn6q" Jul 28 22:22:48.089794 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:22:48.089776 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv46g\" (UniqueName: \"kubernetes.io/projected/e298568e-b131-4a29-b479-d7bc20feed12-kube-api-access-wv46g\") pod \"maas-controller-d6457fd9-bvn6q\" (UID: \"e298568e-b131-4a29-b479-d7bc20feed12\") " pod="opendatahub/maas-controller-d6457fd9-bvn6q" Jul 28 22:22:48.172732 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:22:48.172644 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d6457fd9-bvn6q" Jul 28 22:22:48.305483 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:22:48.305457 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d6457fd9-bvn6q"] Jul 28 22:22:48.307685 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:22:48.307658 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode298568e_b131_4a29_b479_d7bc20feed12.slice/crio-9dfadd33be824a14752f3ce60c544460fae03f159ddcf678118cd13239ad42d5 WatchSource:0}: Error finding container 9dfadd33be824a14752f3ce60c544460fae03f159ddcf678118cd13239ad42d5: Status 404 returned error can't find the container with id 9dfadd33be824a14752f3ce60c544460fae03f159ddcf678118cd13239ad42d5 Jul 28 22:22:48.309071 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:22:48.309051 2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 22:22:48.342086 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:22:48.342061 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d6457fd9-bvn6q" event={"ID":"e298568e-b131-4a29-b479-d7bc20feed12","Type":"ContainerStarted","Data":"9dfadd33be824a14752f3ce60c544460fae03f159ddcf678118cd13239ad42d5"} Jul 28 22:22:49.349222 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:22:49.349128 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d6457fd9-bvn6q" event={"ID":"e298568e-b131-4a29-b479-d7bc20feed12","Type":"ContainerStarted","Data":"433825d52310393bc8513c9b9b75433856bdf9ccfd32501be7880f2e1939de9a"} Jul 28 22:22:49.349222 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:22:49.349193 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-d6457fd9-bvn6q" Jul 28 22:22:49.366966 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:22:49.366912 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-d6457fd9-bvn6q" podStartSLOduration=1.644113325 podStartE2EDuration="2.366893683s" totalImagesPullingTime="722.780358ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:22:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:22:49.364104312 +0000 UTC m=+789.451362382" watchObservedRunningTime="2026-07-28 22:22:49.366893683 +0000 UTC m=+789.454151756" Jul 28 22:23:00.356453 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:23:00.356417 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-d6457fd9-bvn6q" Jul 28 22:23:04.991366 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:23:04.991324 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-d6457fd9-bvn6q"] Jul 28 22:23:04.991890 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:23:04.991520 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-d6457fd9-bvn6q" podUID="e298568e-b131-4a29-b479-d7bc20feed12" containerName="manager" containerID="cri-o://433825d52310393bc8513c9b9b75433856bdf9ccfd32501be7880f2e1939de9a" gracePeriod=30 Jul 28 22:23:05.240794 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:23:05.240769 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d6457fd9-bvn6q" Jul 28 22:23:05.328722 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:23:05.328619 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wv46g\" (UniqueName: \"kubernetes.io/projected/e298568e-b131-4a29-b479-d7bc20feed12-kube-api-access-wv46g\") pod \"e298568e-b131-4a29-b479-d7bc20feed12\" (UID: \"e298568e-b131-4a29-b479-d7bc20feed12\") " Jul 28 22:23:05.328722 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:23:05.328715 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e298568e-b131-4a29-b479-d7bc20feed12-webhook-cert\") pod \"e298568e-b131-4a29-b479-d7bc20feed12\" (UID: \"e298568e-b131-4a29-b479-d7bc20feed12\") " Jul 28 22:23:05.330935 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:23:05.330909 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e298568e-b131-4a29-b479-d7bc20feed12-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "e298568e-b131-4a29-b479-d7bc20feed12" (UID: "e298568e-b131-4a29-b479-d7bc20feed12"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 22:23:05.330935 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:23:05.330915 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e298568e-b131-4a29-b479-d7bc20feed12-kube-api-access-wv46g" (OuterVolumeSpecName: "kube-api-access-wv46g") pod "e298568e-b131-4a29-b479-d7bc20feed12" (UID: "e298568e-b131-4a29-b479-d7bc20feed12"). InnerVolumeSpecName "kube-api-access-wv46g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:23:05.429863 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:23:05.429838 2553 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e298568e-b131-4a29-b479-d7bc20feed12-webhook-cert\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:23:05.429863 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:23:05.429860 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wv46g\" (UniqueName: \"kubernetes.io/projected/e298568e-b131-4a29-b479-d7bc20feed12-kube-api-access-wv46g\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:23:05.433109 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:23:05.433082 2553 generic.go:356] "Generic (PLEG): container finished" podID="e298568e-b131-4a29-b479-d7bc20feed12" containerID="433825d52310393bc8513c9b9b75433856bdf9ccfd32501be7880f2e1939de9a" exitCode=0 Jul 28 22:23:05.433252 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:23:05.433130 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d6457fd9-bvn6q" Jul 28 22:23:05.433252 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:23:05.433159 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d6457fd9-bvn6q" event={"ID":"e298568e-b131-4a29-b479-d7bc20feed12","Type":"ContainerDied","Data":"433825d52310393bc8513c9b9b75433856bdf9ccfd32501be7880f2e1939de9a"} Jul 28 22:23:05.433252 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:23:05.433205 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d6457fd9-bvn6q" event={"ID":"e298568e-b131-4a29-b479-d7bc20feed12","Type":"ContainerDied","Data":"9dfadd33be824a14752f3ce60c544460fae03f159ddcf678118cd13239ad42d5"} Jul 28 22:23:05.433252 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:23:05.433227 2553 scope.go:117] "RemoveContainer" containerID="433825d52310393bc8513c9b9b75433856bdf9ccfd32501be7880f2e1939de9a" Jul 28 22:23:05.442626 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:23:05.442612 2553 scope.go:117] "RemoveContainer" containerID="433825d52310393bc8513c9b9b75433856bdf9ccfd32501be7880f2e1939de9a" Jul 28 22:23:05.442962 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:23:05.442945 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"433825d52310393bc8513c9b9b75433856bdf9ccfd32501be7880f2e1939de9a\": container with ID starting with 433825d52310393bc8513c9b9b75433856bdf9ccfd32501be7880f2e1939de9a not found: ID does not exist" containerID="433825d52310393bc8513c9b9b75433856bdf9ccfd32501be7880f2e1939de9a" Jul 28 22:23:05.443025 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:23:05.442970 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"433825d52310393bc8513c9b9b75433856bdf9ccfd32501be7880f2e1939de9a"} err="failed to get container status \"433825d52310393bc8513c9b9b75433856bdf9ccfd32501be7880f2e1939de9a\": rpc error: code = NotFound desc = could not find container \"433825d52310393bc8513c9b9b75433856bdf9ccfd32501be7880f2e1939de9a\": container with ID starting with 433825d52310393bc8513c9b9b75433856bdf9ccfd32501be7880f2e1939de9a not found: ID does not exist" Jul 28 22:23:05.453904 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:23:05.453884 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-d6457fd9-bvn6q"] Jul 28 22:23:05.457597 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:23:05.457577 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-d6457fd9-bvn6q"] Jul 28 22:23:06.271062 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:23:06.271029 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-d6457fd9-dghbg"] Jul 28 22:23:06.271441 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:23:06.271374 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e298568e-b131-4a29-b479-d7bc20feed12" containerName="manager" Jul 28 22:23:06.271441 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:23:06.271385 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="e298568e-b131-4a29-b479-d7bc20feed12" containerName="manager" Jul 28 22:23:06.271515 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:23:06.271445 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="e298568e-b131-4a29-b479-d7bc20feed12" containerName="manager" Jul 28 22:23:06.275672 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:23:06.275653 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d6457fd9-dghbg" Jul 28 22:23:06.280568 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:23:06.280547 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 28 22:23:06.280732 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:23:06.280677 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 28 22:23:06.281432 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:23:06.281416 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-7k5cv\"" Jul 28 22:23:06.293461 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:23:06.293434 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d6457fd9-dghbg"] Jul 28 22:23:06.337115 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:23:06.337074 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4184e9a3-2f74-4275-8c85-5764766290cd-webhook-cert\") pod \"maas-controller-d6457fd9-dghbg\" (UID: \"4184e9a3-2f74-4275-8c85-5764766290cd\") " pod="opendatahub/maas-controller-d6457fd9-dghbg" Jul 28 22:23:06.337296 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:23:06.337160 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cbmz\" (UniqueName: \"kubernetes.io/projected/4184e9a3-2f74-4275-8c85-5764766290cd-kube-api-access-9cbmz\") pod \"maas-controller-d6457fd9-dghbg\" (UID: \"4184e9a3-2f74-4275-8c85-5764766290cd\") " pod="opendatahub/maas-controller-d6457fd9-dghbg" Jul 28 22:23:06.404717 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:23:06.404674 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e298568e-b131-4a29-b479-d7bc20feed12" path="/var/lib/kubelet/pods/e298568e-b131-4a29-b479-d7bc20feed12/volumes" Jul 28 22:23:06.438505 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:23:06.438474 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4184e9a3-2f74-4275-8c85-5764766290cd-webhook-cert\") pod \"maas-controller-d6457fd9-dghbg\" (UID: \"4184e9a3-2f74-4275-8c85-5764766290cd\") " pod="opendatahub/maas-controller-d6457fd9-dghbg" Jul 28 22:23:06.438693 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:23:06.438571 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cbmz\" (UniqueName: \"kubernetes.io/projected/4184e9a3-2f74-4275-8c85-5764766290cd-kube-api-access-9cbmz\") pod \"maas-controller-d6457fd9-dghbg\" (UID: \"4184e9a3-2f74-4275-8c85-5764766290cd\") " pod="opendatahub/maas-controller-d6457fd9-dghbg" Jul 28 22:23:06.441155 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:23:06.441132 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4184e9a3-2f74-4275-8c85-5764766290cd-webhook-cert\") pod \"maas-controller-d6457fd9-dghbg\" (UID: \"4184e9a3-2f74-4275-8c85-5764766290cd\") " pod="opendatahub/maas-controller-d6457fd9-dghbg" Jul 28 22:23:06.446863 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:23:06.446837 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cbmz\" (UniqueName: \"kubernetes.io/projected/4184e9a3-2f74-4275-8c85-5764766290cd-kube-api-access-9cbmz\") pod \"maas-controller-d6457fd9-dghbg\" (UID: \"4184e9a3-2f74-4275-8c85-5764766290cd\") " pod="opendatahub/maas-controller-d6457fd9-dghbg" Jul 28 22:23:06.585822 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:23:06.585734 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d6457fd9-dghbg" Jul 28 22:23:06.718418 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:23:06.718385 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d6457fd9-dghbg"] Jul 28 22:23:06.720349 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:23:06.720319 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4184e9a3_2f74_4275_8c85_5764766290cd.slice/crio-ace47cd79b39f5d781507fc33e88b62ec3c65a9362c638dcd6bbe8db00812313 WatchSource:0}: Error finding container ace47cd79b39f5d781507fc33e88b62ec3c65a9362c638dcd6bbe8db00812313: Status 404 returned error can't find the container with id ace47cd79b39f5d781507fc33e88b62ec3c65a9362c638dcd6bbe8db00812313 Jul 28 22:23:07.446581 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:23:07.446490 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d6457fd9-dghbg" event={"ID":"4184e9a3-2f74-4275-8c85-5764766290cd","Type":"ContainerStarted","Data":"4094f3f3fc3e80d6fc196002db5acfb172fa049546093e375ca86cebbcb1a783"} Jul 28 22:23:07.446992 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:23:07.446582 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d6457fd9-dghbg" event={"ID":"4184e9a3-2f74-4275-8c85-5764766290cd","Type":"ContainerStarted","Data":"ace47cd79b39f5d781507fc33e88b62ec3c65a9362c638dcd6bbe8db00812313"} Jul 28 22:23:07.446992 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:23:07.446613 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-d6457fd9-dghbg" Jul 28 22:23:07.464268 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:23:07.464213 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-d6457fd9-dghbg" podStartSLOduration=1.074521192 podStartE2EDuration="1.464197914s" totalImagesPullingTime="389.676722ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:23:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:23:07.462007652 +0000 UTC m=+807.549265722" watchObservedRunningTime="2026-07-28 22:23:07.464197914 +0000 UTC m=+807.551455983" Jul 28 22:23:18.454383 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:23:18.454349 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-d6457fd9-dghbg" Jul 28 22:24:40.351454 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:24:40.351379 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mldhp_e1ced996-f136-4d3e-9c85-a5b7809b4517/console-operator/1.log" Jul 28 22:24:40.352314 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:24:40.352295 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mldhp_e1ced996-f136-4d3e-9c85-a5b7809b4517/console-operator/1.log" Jul 28 22:29:40.385318 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:29:40.385291 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mldhp_e1ced996-f136-4d3e-9c85-a5b7809b4517/console-operator/1.log" Jul 28 22:29:40.387425 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:29:40.387401 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mldhp_e1ced996-f136-4d3e-9c85-a5b7809b4517/console-operator/1.log" Jul 28 22:34:40.421985 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:34:40.421958 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mldhp_e1ced996-f136-4d3e-9c85-a5b7809b4517/console-operator/1.log" Jul 28 22:34:40.424989 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:34:40.424959 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mldhp_e1ced996-f136-4d3e-9c85-a5b7809b4517/console-operator/1.log" Jul 28 22:36:11.602237 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:11.602206 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp"] Jul 28 22:36:11.605504 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:11.605488 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp" Jul 28 22:36:11.608013 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:11.607997 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-c67c1cbd-openshift-default-dockercfg-rzrx5\"" Jul 28 22:36:11.625085 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:11.625056 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp"] Jul 28 22:36:11.702915 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:11.702885 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7-istio-podinfo\") pod \"e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp\" (UID: \"2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7\") " pod="openshift-ingress/e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp" Jul 28 22:36:11.703112 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:11.702924 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7-workload-socket\") pod \"e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp\" (UID: \"2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7\") " pod="openshift-ingress/e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp" Jul 28 22:36:11.703112 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:11.702946 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7-istio-token\") pod \"e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp\" (UID: \"2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7\") " pod="openshift-ingress/e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp" Jul 28 22:36:11.703112 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:11.702962 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7-istiod-ca-cert\") pod \"e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp\" (UID: \"2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7\") " pod="openshift-ingress/e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp" Jul 28 22:36:11.703297 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:11.703108 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7-workload-certs\") pod \"e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp\" (UID: \"2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7\") " pod="openshift-ingress/e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp" Jul 28 22:36:11.703297 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:11.703184 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djjb8\" (UniqueName: \"kubernetes.io/projected/2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7-kube-api-access-djjb8\") pod \"e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp\" (UID: \"2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7\") " pod="openshift-ingress/e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp" Jul 28 22:36:11.703297 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:11.703225 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7-istio-data\") pod \"e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp\" (UID: \"2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7\") " pod="openshift-ingress/e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp" Jul 28 22:36:11.703297 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:11.703251 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7-credential-socket\") pod \"e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp\" (UID: \"2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7\") " pod="openshift-ingress/e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp" Jul 28 22:36:11.703297 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:11.703287 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7-istio-envoy\") pod \"e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp\" (UID: \"2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7\") " pod="openshift-ingress/e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp" Jul 28 22:36:11.804501 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:11.804464 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djjb8\" (UniqueName: \"kubernetes.io/projected/2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7-kube-api-access-djjb8\") pod \"e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp\" (UID: \"2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7\") " pod="openshift-ingress/e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp" Jul 28 22:36:11.804501 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:11.804507 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7-istio-data\") pod \"e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp\" (UID: \"2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7\") " pod="openshift-ingress/e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp" Jul 28 22:36:11.804794 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:11.804526 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7-credential-socket\") pod \"e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp\" (UID: \"2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7\") " pod="openshift-ingress/e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp" Jul 28 22:36:11.804794 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:11.804650 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7-istio-envoy\") pod \"e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp\" (UID: \"2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7\") " pod="openshift-ingress/e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp" Jul 28 22:36:11.804794 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:11.804762 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7-istio-podinfo\") pod \"e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp\" (UID: \"2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7\") " pod="openshift-ingress/e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp" Jul 28 22:36:11.804794 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:11.804793 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7-workload-socket\") pod \"e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp\" (UID: \"2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7\") " pod="openshift-ingress/e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp" Jul 28 22:36:11.805017 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:11.804816 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7-istio-token\") pod \"e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp\" (UID: \"2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7\") " pod="openshift-ingress/e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp" Jul 28 22:36:11.805017 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:11.804840 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7-istiod-ca-cert\") pod \"e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp\" (UID: \"2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7\") " pod="openshift-ingress/e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp" Jul 28 22:36:11.805017 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:11.804898 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7-workload-certs\") pod \"e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp\" (UID: \"2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7\") " pod="openshift-ingress/e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp" Jul 28 22:36:11.805017 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:11.804922 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7-credential-socket\") pod \"e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp\" (UID: \"2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7\") " pod="openshift-ingress/e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp" Jul 28 22:36:11.805218 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:11.805131 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7-workload-socket\") pod \"e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp\" (UID: \"2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7\") " pod="openshift-ingress/e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp" Jul 28 22:36:11.805274 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:11.805227 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7-istio-data\") pod \"e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp\" (UID: \"2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7\") " pod="openshift-ingress/e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp" Jul 28 22:36:11.805274 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:11.805247 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7-workload-certs\") pod \"e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp\" (UID: \"2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7\") " pod="openshift-ingress/e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp" Jul 28 22:36:11.805480 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:11.805453 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7-istiod-ca-cert\") pod \"e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp\" (UID: \"2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7\") " pod="openshift-ingress/e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp" Jul 28 22:36:11.807080 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:11.807062 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7-istio-envoy\") pod \"e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp\" (UID: \"2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7\") " pod="openshift-ingress/e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp" Jul 28 22:36:11.807509 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:11.807484 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7-istio-podinfo\") pod \"e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp\" (UID: \"2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7\") " pod="openshift-ingress/e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp" Jul 28 22:36:11.812856 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:11.812829 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7-istio-token\") pod \"e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp\" (UID: \"2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7\") " pod="openshift-ingress/e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp" Jul 28 22:36:11.813014 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:11.812996 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djjb8\" (UniqueName: \"kubernetes.io/projected/2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7-kube-api-access-djjb8\") pod \"e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp\" (UID: \"2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7\") " pod="openshift-ingress/e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp" Jul 28 22:36:11.915781 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:11.915671 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp" Jul 28 22:36:12.054363 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:12.054307 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp"] Jul 28 22:36:12.056661 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:36:12.056630 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dca4ce7_8b09_4a9b_9c17_4026f95cb0f7.slice/crio-4769c6716c7c569c9cad59ac68e39e18be69739c080d4d64aa82b120ab70ddd4 WatchSource:0}: Error finding container 4769c6716c7c569c9cad59ac68e39e18be69739c080d4d64aa82b120ab70ddd4: Status 404 returned error can't find the container with id 4769c6716c7c569c9cad59ac68e39e18be69739c080d4d64aa82b120ab70ddd4 Jul 28 22:36:12.058922 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:12.058892 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:36:12.059001 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:12.058955 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:36:12.059001 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:12.058984 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:36:12.436482 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:12.436446 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp" event={"ID":"2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7","Type":"ContainerStarted","Data":"dbfb822e360ced9bf6d1482d686464cd7bb498429449ea542367719278e0e87b"} Jul 28 22:36:12.436695 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:12.436512 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp" event={"ID":"2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7","Type":"ContainerStarted","Data":"4769c6716c7c569c9cad59ac68e39e18be69739c080d4d64aa82b120ab70ddd4"} Jul 28 22:36:12.455662 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:12.455610 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp" podStartSLOduration=1.455589676 podStartE2EDuration="1.455589676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:36:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:36:12.453891655 +0000 UTC m=+1592.541149748" watchObservedRunningTime="2026-07-28 22:36:12.455589676 +0000 UTC m=+1592.542847809" Jul 28 22:36:12.915918 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:12.915869 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp" Jul 28 22:36:13.922252 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:13.922225 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp" Jul 28 22:36:14.118952 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:14.118917 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c67c1cbd-d4997f4b7-tfk6w"] Jul 28 22:36:14.122167 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:14.122144 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c67c1cbd-d4997f4b7-tfk6w" Jul 28 22:36:14.125140 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:14.125117 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-c67c1cbd-serving-cert\"" Jul 28 22:36:14.137084 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:14.137057 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c67c1cbd-d4997f4b7-tfk6w"] Jul 28 22:36:14.224977 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:14.224946 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/756bc0db-1a9d-4035-babd-661d35b79ae0-openshift-service-ca\") pod \"maas-api-e2e-ait-c67c1cbd-d4997f4b7-tfk6w\" (UID: \"756bc0db-1a9d-4035-babd-661d35b79ae0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c67c1cbd-d4997f4b7-tfk6w" Jul 28 22:36:14.225140 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:14.225018 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9msb8\" (UniqueName: \"kubernetes.io/projected/756bc0db-1a9d-4035-babd-661d35b79ae0-kube-api-access-9msb8\") pod \"maas-api-e2e-ait-c67c1cbd-d4997f4b7-tfk6w\" (UID: \"756bc0db-1a9d-4035-babd-661d35b79ae0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c67c1cbd-d4997f4b7-tfk6w" Jul 28 22:36:14.225140 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:14.225094 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/756bc0db-1a9d-4035-babd-661d35b79ae0-maas-api-tls\") pod \"maas-api-e2e-ait-c67c1cbd-d4997f4b7-tfk6w\" (UID: \"756bc0db-1a9d-4035-babd-661d35b79ae0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c67c1cbd-d4997f4b7-tfk6w" Jul 28 22:36:14.326394 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:14.326365 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9msb8\" (UniqueName: \"kubernetes.io/projected/756bc0db-1a9d-4035-babd-661d35b79ae0-kube-api-access-9msb8\") pod \"maas-api-e2e-ait-c67c1cbd-d4997f4b7-tfk6w\" (UID: \"756bc0db-1a9d-4035-babd-661d35b79ae0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c67c1cbd-d4997f4b7-tfk6w" Jul 28 22:36:14.326572 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:14.326416 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/756bc0db-1a9d-4035-babd-661d35b79ae0-maas-api-tls\") pod \"maas-api-e2e-ait-c67c1cbd-d4997f4b7-tfk6w\" (UID: \"756bc0db-1a9d-4035-babd-661d35b79ae0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c67c1cbd-d4997f4b7-tfk6w" Jul 28 22:36:14.326572 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:14.326457 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/756bc0db-1a9d-4035-babd-661d35b79ae0-openshift-service-ca\") pod \"maas-api-e2e-ait-c67c1cbd-d4997f4b7-tfk6w\" (UID: \"756bc0db-1a9d-4035-babd-661d35b79ae0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c67c1cbd-d4997f4b7-tfk6w" Jul 28 22:36:14.326654 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:36:14.326571 2553 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-c67c1cbd-serving-cert: secret "maas-api-e2e-ait-c67c1cbd-serving-cert" not found Jul 28 22:36:14.326654 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:36:14.326638 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/756bc0db-1a9d-4035-babd-661d35b79ae0-maas-api-tls podName:756bc0db-1a9d-4035-babd-661d35b79ae0 nodeName:}" failed. No retries permitted until 2026-07-28 22:36:14.826618754 +0000 UTC m=+1594.913876809 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/756bc0db-1a9d-4035-babd-661d35b79ae0-maas-api-tls") pod "maas-api-e2e-ait-c67c1cbd-d4997f4b7-tfk6w" (UID: "756bc0db-1a9d-4035-babd-661d35b79ae0") : secret "maas-api-e2e-ait-c67c1cbd-serving-cert" not found Jul 28 22:36:14.327096 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:14.327080 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/756bc0db-1a9d-4035-babd-661d35b79ae0-openshift-service-ca\") pod \"maas-api-e2e-ait-c67c1cbd-d4997f4b7-tfk6w\" (UID: \"756bc0db-1a9d-4035-babd-661d35b79ae0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c67c1cbd-d4997f4b7-tfk6w" Jul 28 22:36:14.337684 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:14.337658 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9msb8\" (UniqueName: \"kubernetes.io/projected/756bc0db-1a9d-4035-babd-661d35b79ae0-kube-api-access-9msb8\") pod \"maas-api-e2e-ait-c67c1cbd-d4997f4b7-tfk6w\" (UID: \"756bc0db-1a9d-4035-babd-661d35b79ae0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c67c1cbd-d4997f4b7-tfk6w" Jul 28 22:36:14.447954 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:14.447928 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp" Jul 28 22:36:14.449369 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:14.449336 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp" Jul 28 22:36:14.830412 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:14.830378 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/756bc0db-1a9d-4035-babd-661d35b79ae0-maas-api-tls\") pod \"maas-api-e2e-ait-c67c1cbd-d4997f4b7-tfk6w\" (UID: \"756bc0db-1a9d-4035-babd-661d35b79ae0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c67c1cbd-d4997f4b7-tfk6w" Jul 28 22:36:14.833218 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:14.833191 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/756bc0db-1a9d-4035-babd-661d35b79ae0-maas-api-tls\") pod \"maas-api-e2e-ait-c67c1cbd-d4997f4b7-tfk6w\" (UID: \"756bc0db-1a9d-4035-babd-661d35b79ae0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c67c1cbd-d4997f4b7-tfk6w" Jul 28 22:36:15.032376 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:15.032346 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c67c1cbd-d4997f4b7-tfk6w" Jul 28 22:36:15.178319 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:15.178291 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c67c1cbd-d4997f4b7-tfk6w"] Jul 28 22:36:15.180198 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:36:15.180172 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod756bc0db_1a9d_4035_babd_661d35b79ae0.slice/crio-6884ea01541cccde8efdd8e540d1a6abe8a9e3cd874cf5a4bd86417f135204d9 WatchSource:0}: Error finding container 6884ea01541cccde8efdd8e540d1a6abe8a9e3cd874cf5a4bd86417f135204d9: Status 404 returned error can't find the container with id 6884ea01541cccde8efdd8e540d1a6abe8a9e3cd874cf5a4bd86417f135204d9 Jul 28 22:36:15.181631 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:15.181609 2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 22:36:15.453642 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:15.453552 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c67c1cbd-d4997f4b7-tfk6w" event={"ID":"756bc0db-1a9d-4035-babd-661d35b79ae0","Type":"ContainerStarted","Data":"6884ea01541cccde8efdd8e540d1a6abe8a9e3cd874cf5a4bd86417f135204d9"} Jul 28 22:36:16.459970 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:16.459935 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c67c1cbd-d4997f4b7-tfk6w" event={"ID":"756bc0db-1a9d-4035-babd-661d35b79ae0","Type":"ContainerStarted","Data":"cda8208016643536bfafdfc95aac96f85e5d7f853cfb13eb52d52fb232141c13"} Jul 28 22:36:16.460416 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:16.460003 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c67c1cbd-d4997f4b7-tfk6w" Jul 28 22:36:16.478214 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:16.478165 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c67c1cbd-d4997f4b7-tfk6w" podStartSLOduration=1.990497856 podStartE2EDuration="2.478151277s" totalImagesPullingTime="487.653421ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:36:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:36:16.475090118 +0000 UTC m=+1596.562348191" watchObservedRunningTime="2026-07-28 22:36:16.478151277 +0000 UTC m=+1596.565409384" Jul 28 22:36:22.471158 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:22.471128 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c67c1cbd-d4997f4b7-tfk6w" Jul 28 22:36:27.519542 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:27.519506 2553 generic.go:356] "Generic (PLEG): container finished" podID="2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7" containerID="dbfb822e360ced9bf6d1482d686464cd7bb498429449ea542367719278e0e87b" exitCode=0 Jul 28 22:36:27.520054 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:27.519579 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp" event={"ID":"2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7","Type":"ContainerDied","Data":"dbfb822e360ced9bf6d1482d686464cd7bb498429449ea542367719278e0e87b"} Jul 28 22:36:27.520054 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:27.519992 2553 scope.go:117] "RemoveContainer" containerID="dbfb822e360ced9bf6d1482d686464cd7bb498429449ea542367719278e0e87b" Jul 28 22:36:27.521034 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:27.521011 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:36:27.521101 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:27.521078 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:36:27.521139 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:27.521109 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:36:28.530970 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:28.530921 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp" event={"ID":"2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7","Type":"ContainerStarted","Data":"bcfeab154551250494639c78b70b5d0edfe24d617099df4fb32c98fe52f98668"} Jul 28 22:36:28.918069 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:28.916898 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp" Jul 28 22:36:28.918409 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:28.918382 2553 patch_prober.go:28] interesting pod/e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Jul 28 22:36:28.918606 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:28.918579 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp" podUID="2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Jul 28 22:36:29.448373 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:29.448322 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp" Jul 28 22:36:29.917052 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:29.916582 2553 patch_prober.go:28] interesting pod/e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Jul 28 22:36:29.917052 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:29.916643 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp" podUID="2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Jul 28 22:36:30.916783 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:30.916670 2553 patch_prober.go:28] interesting pod/e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Jul 28 22:36:30.916783 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:30.916756 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp" podUID="2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Jul 28 22:36:31.917012 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:31.916969 2553 patch_prober.go:28] interesting pod/e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Jul 28 22:36:31.917534 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:31.917054 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp" podUID="2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Jul 28 22:36:32.920546 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:32.920515 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp" Jul 28 22:36:32.921515 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:32.921494 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp" Jul 28 22:36:44.696903 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:44.696866 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c67c1cbd-d4997f4b7-tfk6w"] Jul 28 22:36:44.697768 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:44.697150 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c67c1cbd-d4997f4b7-tfk6w" podUID="756bc0db-1a9d-4035-babd-661d35b79ae0" containerName="maas-api" containerID="cri-o://cda8208016643536bfafdfc95aac96f85e5d7f853cfb13eb52d52fb232141c13" gracePeriod=30 Jul 28 22:36:44.976052 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:44.976030 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c67c1cbd-d4997f4b7-tfk6w" Jul 28 22:36:45.008190 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:45.008128 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9msb8\" (UniqueName: \"kubernetes.io/projected/756bc0db-1a9d-4035-babd-661d35b79ae0-kube-api-access-9msb8\") pod \"756bc0db-1a9d-4035-babd-661d35b79ae0\" (UID: \"756bc0db-1a9d-4035-babd-661d35b79ae0\") " Jul 28 22:36:45.008371 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:45.008217 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/756bc0db-1a9d-4035-babd-661d35b79ae0-openshift-service-ca\") pod \"756bc0db-1a9d-4035-babd-661d35b79ae0\" (UID: \"756bc0db-1a9d-4035-babd-661d35b79ae0\") " Jul 28 22:36:45.008371 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:45.008248 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/756bc0db-1a9d-4035-babd-661d35b79ae0-maas-api-tls\") pod \"756bc0db-1a9d-4035-babd-661d35b79ae0\" (UID: \"756bc0db-1a9d-4035-babd-661d35b79ae0\") " Jul 28 22:36:45.008637 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:45.008592 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/756bc0db-1a9d-4035-babd-661d35b79ae0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "756bc0db-1a9d-4035-babd-661d35b79ae0" (UID: "756bc0db-1a9d-4035-babd-661d35b79ae0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 22:36:45.012096 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:45.012070 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/756bc0db-1a9d-4035-babd-661d35b79ae0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "756bc0db-1a9d-4035-babd-661d35b79ae0" (UID: "756bc0db-1a9d-4035-babd-661d35b79ae0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 22:36:45.012188 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:45.012091 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/756bc0db-1a9d-4035-babd-661d35b79ae0-kube-api-access-9msb8" (OuterVolumeSpecName: "kube-api-access-9msb8") pod "756bc0db-1a9d-4035-babd-661d35b79ae0" (UID: "756bc0db-1a9d-4035-babd-661d35b79ae0"). InnerVolumeSpecName "kube-api-access-9msb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:36:45.023960 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:45.023931 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp"] Jul 28 22:36:45.024220 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:45.024190 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp" podUID="2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7" containerName="istio-proxy" containerID="cri-o://bcfeab154551250494639c78b70b5d0edfe24d617099df4fb32c98fe52f98668" gracePeriod=30 Jul 28 22:36:45.109052 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:45.109014 2553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/756bc0db-1a9d-4035-babd-661d35b79ae0-openshift-service-ca\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:36:45.109052 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:45.109043 2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/756bc0db-1a9d-4035-babd-661d35b79ae0-maas-api-tls\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:36:45.109298 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:45.109053 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9msb8\" (UniqueName: \"kubernetes.io/projected/756bc0db-1a9d-4035-babd-661d35b79ae0-kube-api-access-9msb8\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:36:45.622885 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:45.622850 2553 generic.go:356] "Generic (PLEG): container finished" podID="756bc0db-1a9d-4035-babd-661d35b79ae0" containerID="cda8208016643536bfafdfc95aac96f85e5d7f853cfb13eb52d52fb232141c13" exitCode=0 Jul 28 22:36:45.623083 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:45.622901 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c67c1cbd-d4997f4b7-tfk6w" Jul 28 22:36:45.623083 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:45.622953 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c67c1cbd-d4997f4b7-tfk6w" event={"ID":"756bc0db-1a9d-4035-babd-661d35b79ae0","Type":"ContainerDied","Data":"cda8208016643536bfafdfc95aac96f85e5d7f853cfb13eb52d52fb232141c13"} Jul 28 22:36:45.623083 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:45.623016 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c67c1cbd-d4997f4b7-tfk6w" event={"ID":"756bc0db-1a9d-4035-babd-661d35b79ae0","Type":"ContainerDied","Data":"6884ea01541cccde8efdd8e540d1a6abe8a9e3cd874cf5a4bd86417f135204d9"} Jul 28 22:36:45.623083 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:45.623042 2553 scope.go:117] "RemoveContainer" containerID="cda8208016643536bfafdfc95aac96f85e5d7f853cfb13eb52d52fb232141c13" Jul 28 22:36:45.632423 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:45.632402 2553 scope.go:117] "RemoveContainer" containerID="cda8208016643536bfafdfc95aac96f85e5d7f853cfb13eb52d52fb232141c13" Jul 28 22:36:45.632667 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:36:45.632647 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cda8208016643536bfafdfc95aac96f85e5d7f853cfb13eb52d52fb232141c13\": container with ID starting with cda8208016643536bfafdfc95aac96f85e5d7f853cfb13eb52d52fb232141c13 not found: ID does not exist" containerID="cda8208016643536bfafdfc95aac96f85e5d7f853cfb13eb52d52fb232141c13" Jul 28 22:36:45.632745 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:45.632675 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cda8208016643536bfafdfc95aac96f85e5d7f853cfb13eb52d52fb232141c13"} err="failed to get container status \"cda8208016643536bfafdfc95aac96f85e5d7f853cfb13eb52d52fb232141c13\": rpc error: code = NotFound desc = could not find container \"cda8208016643536bfafdfc95aac96f85e5d7f853cfb13eb52d52fb232141c13\": container with ID starting with cda8208016643536bfafdfc95aac96f85e5d7f853cfb13eb52d52fb232141c13 not found: ID does not exist" Jul 28 22:36:45.643393 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:45.643368 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c67c1cbd-d4997f4b7-tfk6w"] Jul 28 22:36:45.654601 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:45.654579 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c67c1cbd-d4997f4b7-tfk6w"] Jul 28 22:36:46.027740 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:46.027692 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c"] Jul 28 22:36:46.028132 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:46.028084 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="756bc0db-1a9d-4035-babd-661d35b79ae0" containerName="maas-api" Jul 28 22:36:46.028132 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:46.028096 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="756bc0db-1a9d-4035-babd-661d35b79ae0" containerName="maas-api" Jul 28 22:36:46.028215 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:46.028158 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="756bc0db-1a9d-4035-babd-661d35b79ae0" containerName="maas-api" Jul 28 22:36:46.032470 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:46.032455 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c" Jul 28 22:36:46.034879 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:46.034852 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-6ad9588a-gw-openshift-default-dockercfg-dzq7q\"" Jul 28 22:36:46.043311 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:46.043287 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c"] Jul 28 22:36:46.116420 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:46.116382 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abfdf350-2744-4709-989d-9943916f2405-istio-envoy\") pod \"e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c\" (UID: \"abfdf350-2744-4709-989d-9943916f2405\") " pod="openshift-ingress/e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c" Jul 28 22:36:46.116603 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:46.116463 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abfdf350-2744-4709-989d-9943916f2405-credential-socket\") pod \"e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c\" (UID: \"abfdf350-2744-4709-989d-9943916f2405\") " pod="openshift-ingress/e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c" Jul 28 22:36:46.116603 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:46.116516 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abfdf350-2744-4709-989d-9943916f2405-istio-podinfo\") pod \"e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c\" (UID: \"abfdf350-2744-4709-989d-9943916f2405\") " pod="openshift-ingress/e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c" Jul 28 22:36:46.116720 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:46.116604 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abfdf350-2744-4709-989d-9943916f2405-istio-token\") pod \"e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c\" (UID: \"abfdf350-2744-4709-989d-9943916f2405\") " pod="openshift-ingress/e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c" Jul 28 22:36:46.116720 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:46.116653 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abfdf350-2744-4709-989d-9943916f2405-workload-certs\") pod \"e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c\" (UID: \"abfdf350-2744-4709-989d-9943916f2405\") " pod="openshift-ingress/e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c" Jul 28 22:36:46.116856 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:46.116749 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abfdf350-2744-4709-989d-9943916f2405-workload-socket\") pod \"e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c\" (UID: \"abfdf350-2744-4709-989d-9943916f2405\") " pod="openshift-ingress/e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c" Jul 28 22:36:46.116856 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:46.116807 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw2vj\" (UniqueName: \"kubernetes.io/projected/abfdf350-2744-4709-989d-9943916f2405-kube-api-access-kw2vj\") pod \"e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c\" (UID: \"abfdf350-2744-4709-989d-9943916f2405\") " pod="openshift-ingress/e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c" Jul 28 22:36:46.116856 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:46.116837 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abfdf350-2744-4709-989d-9943916f2405-istiod-ca-cert\") pod \"e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c\" (UID: \"abfdf350-2744-4709-989d-9943916f2405\") " pod="openshift-ingress/e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c" Jul 28 22:36:46.116994 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:46.116875 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abfdf350-2744-4709-989d-9943916f2405-istio-data\") pod \"e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c\" (UID: \"abfdf350-2744-4709-989d-9943916f2405\") " pod="openshift-ingress/e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c" Jul 28 22:36:46.217986 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:46.217952 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abfdf350-2744-4709-989d-9943916f2405-istio-token\") pod \"e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c\" (UID: \"abfdf350-2744-4709-989d-9943916f2405\") " pod="openshift-ingress/e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c" Jul 28 22:36:46.218170 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:46.217998 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abfdf350-2744-4709-989d-9943916f2405-workload-certs\") pod \"e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c\" (UID: \"abfdf350-2744-4709-989d-9943916f2405\") " pod="openshift-ingress/e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c" Jul 28 22:36:46.218170 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:46.218032 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abfdf350-2744-4709-989d-9943916f2405-workload-socket\") pod \"e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c\" (UID: \"abfdf350-2744-4709-989d-9943916f2405\") " pod="openshift-ingress/e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c" Jul 28 22:36:46.218256 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:46.218228 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw2vj\" (UniqueName: \"kubernetes.io/projected/abfdf350-2744-4709-989d-9943916f2405-kube-api-access-kw2vj\") pod \"e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c\" (UID: \"abfdf350-2744-4709-989d-9943916f2405\") " pod="openshift-ingress/e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c" Jul 28 22:36:46.218317 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:46.218276 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abfdf350-2744-4709-989d-9943916f2405-istiod-ca-cert\") pod \"e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c\" (UID: \"abfdf350-2744-4709-989d-9943916f2405\") " pod="openshift-ingress/e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c" Jul 28 22:36:46.218375 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:46.218325 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abfdf350-2744-4709-989d-9943916f2405-istio-data\") pod \"e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c\" (UID: \"abfdf350-2744-4709-989d-9943916f2405\") " pod="openshift-ingress/e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c" Jul 28 22:36:46.218434 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:46.218371 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abfdf350-2744-4709-989d-9943916f2405-istio-envoy\") pod \"e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c\" (UID: \"abfdf350-2744-4709-989d-9943916f2405\") " pod="openshift-ingress/e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c" Jul 28 22:36:46.218434 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:46.218416 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abfdf350-2744-4709-989d-9943916f2405-workload-socket\") pod \"e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c\" (UID: \"abfdf350-2744-4709-989d-9943916f2405\") " pod="openshift-ingress/e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c" Jul 28 22:36:46.218538 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:46.218434 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abfdf350-2744-4709-989d-9943916f2405-credential-socket\") pod \"e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c\" (UID: \"abfdf350-2744-4709-989d-9943916f2405\") " pod="openshift-ingress/e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c" Jul 28 22:36:46.218538 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:46.218450 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abfdf350-2744-4709-989d-9943916f2405-workload-certs\") pod \"e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c\" (UID: \"abfdf350-2744-4709-989d-9943916f2405\") " pod="openshift-ingress/e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c" Jul 28 22:36:46.218538 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:46.218476 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abfdf350-2744-4709-989d-9943916f2405-istio-podinfo\") pod \"e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c\" (UID: \"abfdf350-2744-4709-989d-9943916f2405\") " pod="openshift-ingress/e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c" Jul 28 22:36:46.218774 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:46.218753 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abfdf350-2744-4709-989d-9943916f2405-istio-data\") pod \"e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c\" (UID: \"abfdf350-2744-4709-989d-9943916f2405\") " pod="openshift-ingress/e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c" Jul 28 22:36:46.218925 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:46.218909 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abfdf350-2744-4709-989d-9943916f2405-credential-socket\") pod \"e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c\" (UID: \"abfdf350-2744-4709-989d-9943916f2405\") " pod="openshift-ingress/e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c" Jul 28 22:36:46.219011 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:46.218965 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abfdf350-2744-4709-989d-9943916f2405-istiod-ca-cert\") pod \"e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c\" (UID: \"abfdf350-2744-4709-989d-9943916f2405\") " pod="openshift-ingress/e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c" Jul 28 22:36:46.220955 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:46.220935 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abfdf350-2744-4709-989d-9943916f2405-istio-envoy\") pod \"e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c\" (UID: \"abfdf350-2744-4709-989d-9943916f2405\") " pod="openshift-ingress/e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c" Jul 28 22:36:46.221158 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:46.221139 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abfdf350-2744-4709-989d-9943916f2405-istio-podinfo\") pod \"e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c\" (UID: \"abfdf350-2744-4709-989d-9943916f2405\") " pod="openshift-ingress/e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c" Jul 28 22:36:46.225238 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:46.225210 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abfdf350-2744-4709-989d-9943916f2405-istio-token\") pod \"e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c\" (UID: \"abfdf350-2744-4709-989d-9943916f2405\") " pod="openshift-ingress/e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c" Jul 28 22:36:46.225349 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:46.225333 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw2vj\" (UniqueName: \"kubernetes.io/projected/abfdf350-2744-4709-989d-9943916f2405-kube-api-access-kw2vj\") pod \"e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c\" (UID: \"abfdf350-2744-4709-989d-9943916f2405\") " pod="openshift-ingress/e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c" Jul 28 22:36:46.343565 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:46.343476 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c" Jul 28 22:36:46.409360 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:46.409328 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="756bc0db-1a9d-4035-babd-661d35b79ae0" path="/var/lib/kubelet/pods/756bc0db-1a9d-4035-babd-661d35b79ae0/volumes" Jul 28 22:36:46.483365 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:46.483329 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c"] Jul 28 22:36:46.485591 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:36:46.485562 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabfdf350_2744_4709_989d_9943916f2405.slice/crio-471edf0f934a7e157c1d2281336ade87baacd03fdd660c36ebb0b04930bb740a WatchSource:0}: Error finding container 471edf0f934a7e157c1d2281336ade87baacd03fdd660c36ebb0b04930bb740a: Status 404 returned error can't find the container with id 471edf0f934a7e157c1d2281336ade87baacd03fdd660c36ebb0b04930bb740a Jul 28 22:36:46.487860 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:46.487833 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:36:46.487917 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:46.487896 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:36:46.487961 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:46.487925 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:36:46.637651 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:46.637575 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c" event={"ID":"abfdf350-2744-4709-989d-9943916f2405","Type":"ContainerStarted","Data":"c394744f04bf303be6102d14adec3c91e1b7b9dc5dd4b8dcde957599114e026f"} Jul 28 22:36:46.637651 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:46.637649 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c" event={"ID":"abfdf350-2744-4709-989d-9943916f2405","Type":"ContainerStarted","Data":"471edf0f934a7e157c1d2281336ade87baacd03fdd660c36ebb0b04930bb740a"} Jul 28 22:36:46.656570 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:46.656510 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c" podStartSLOduration=0.656488982 podStartE2EDuration="656.488982ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:36:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:36:46.654451589 +0000 UTC m=+1626.741709673" watchObservedRunningTime="2026-07-28 22:36:46.656488982 +0000 UTC m=+1626.743747052" Jul 28 22:36:47.344194 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:47.344154 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c" Jul 28 22:36:47.348842 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:47.348817 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c" Jul 28 22:36:47.643510 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:47.643429 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c" Jul 28 22:36:47.644378 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:47.644356 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c" Jul 28 22:36:47.923782 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:47.923653 2553 patch_prober.go:28] interesting pod/e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Jul 28 22:36:47.923782 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:47.923750 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp" podUID="2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 22:36:50.284217 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:50.284191 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp" Jul 28 22:36:50.357923 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:50.357841 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7-istio-data\") pod \"2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7\" (UID: \"2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7\") " Jul 28 22:36:50.357923 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:50.357910 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7-credential-socket\") pod \"2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7\" (UID: \"2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7\") " Jul 28 22:36:50.358148 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:50.357940 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7-istio-token\") pod \"2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7\" (UID: \"2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7\") " Jul 28 22:36:50.358148 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:50.357962 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7-istio-envoy\") pod \"2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7\" (UID: \"2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7\") " Jul 28 22:36:50.358148 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:50.357993 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7-workload-certs\") pod \"2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7\" (UID: \"2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7\") " Jul 28 22:36:50.358148 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:50.358020 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7-istio-podinfo\") pod \"2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7\" (UID: \"2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7\") " Jul 28 22:36:50.358148 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:50.358082 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7-workload-socket\") pod \"2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7\" (UID: \"2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7\") " Jul 28 22:36:50.358148 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:50.358120 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7-istiod-ca-cert\") pod \"2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7\" (UID: \"2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7\") " Jul 28 22:36:50.358453 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:50.358164 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-djjb8\" (UniqueName: \"kubernetes.io/projected/2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7-kube-api-access-djjb8\") pod \"2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7\" (UID: \"2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7\") " Jul 28 22:36:50.358453 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:50.358219 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7" (UID: "2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:36:50.358554 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:50.358494 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7" (UID: "2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:36:50.358554 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:50.358509 2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7-credential-socket\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:36:50.358657 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:50.358594 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7" (UID: "2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:36:50.359062 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:50.359036 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7" (UID: "2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 22:36:50.359062 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:50.359045 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7-istio-data" (OuterVolumeSpecName: "istio-data") pod "2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7" (UID: "2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:36:50.360462 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:50.360434 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7-kube-api-access-djjb8" (OuterVolumeSpecName: "kube-api-access-djjb8") pod "2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7" (UID: "2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7"). InnerVolumeSpecName "kube-api-access-djjb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:36:50.360571 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:50.360478 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7-istio-token" (OuterVolumeSpecName: "istio-token") pod "2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7" (UID: "2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:36:50.360571 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:50.360528 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7" (UID: "2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 22:36:50.360951 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:50.360936 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7" (UID: "2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:36:50.459211 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:50.459182 2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7-workload-socket\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:36:50.459211 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:50.459208 2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7-istiod-ca-cert\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:36:50.459211 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:50.459218 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djjb8\" (UniqueName: \"kubernetes.io/projected/2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7-kube-api-access-djjb8\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:36:50.459548 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:50.459227 2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7-istio-data\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:36:50.459548 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:50.459237 2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7-istio-token\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:36:50.459548 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:50.459245 2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7-istio-envoy\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:36:50.459548 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:50.459253 2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7-workload-certs\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:36:50.459548 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:50.459261 2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7-istio-podinfo\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:36:50.662482 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:50.662388 2553 generic.go:356] "Generic (PLEG): container finished" podID="2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7" containerID="bcfeab154551250494639c78b70b5d0edfe24d617099df4fb32c98fe52f98668" exitCode=0 Jul 28 22:36:50.662482 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:50.662447 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp" Jul 28 22:36:50.662482 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:50.662471 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp" event={"ID":"2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7","Type":"ContainerDied","Data":"bcfeab154551250494639c78b70b5d0edfe24d617099df4fb32c98fe52f98668"} Jul 28 22:36:50.662828 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:50.662523 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp" event={"ID":"2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7","Type":"ContainerDied","Data":"4769c6716c7c569c9cad59ac68e39e18be69739c080d4d64aa82b120ab70ddd4"} Jul 28 22:36:50.662828 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:50.662544 2553 scope.go:117] "RemoveContainer" containerID="bcfeab154551250494639c78b70b5d0edfe24d617099df4fb32c98fe52f98668" Jul 28 22:36:50.672167 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:50.672147 2553 scope.go:117] "RemoveContainer" containerID="dbfb822e360ced9bf6d1482d686464cd7bb498429449ea542367719278e0e87b" Jul 28 22:36:50.678405 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:50.678381 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp"] Jul 28 22:36:50.682014 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:50.681884 2553 scope.go:117] "RemoveContainer" containerID="bcfeab154551250494639c78b70b5d0edfe24d617099df4fb32c98fe52f98668" Jul 28 22:36:50.682410 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:36:50.682319 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcfeab154551250494639c78b70b5d0edfe24d617099df4fb32c98fe52f98668\": container with ID starting with bcfeab154551250494639c78b70b5d0edfe24d617099df4fb32c98fe52f98668 not found: ID does not exist" containerID="bcfeab154551250494639c78b70b5d0edfe24d617099df4fb32c98fe52f98668" Jul 28 22:36:50.682410 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:50.682356 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcfeab154551250494639c78b70b5d0edfe24d617099df4fb32c98fe52f98668"} err="failed to get container status \"bcfeab154551250494639c78b70b5d0edfe24d617099df4fb32c98fe52f98668\": rpc error: code = NotFound desc = could not find container \"bcfeab154551250494639c78b70b5d0edfe24d617099df4fb32c98fe52f98668\": container with ID starting with bcfeab154551250494639c78b70b5d0edfe24d617099df4fb32c98fe52f98668 not found: ID does not exist" Jul 28 22:36:50.682410 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:50.682384 2553 scope.go:117] "RemoveContainer" containerID="dbfb822e360ced9bf6d1482d686464cd7bb498429449ea542367719278e0e87b" Jul 28 22:36:50.682776 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:36:50.682737 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbfb822e360ced9bf6d1482d686464cd7bb498429449ea542367719278e0e87b\": container with ID starting with dbfb822e360ced9bf6d1482d686464cd7bb498429449ea542367719278e0e87b not found: ID does not exist" containerID="dbfb822e360ced9bf6d1482d686464cd7bb498429449ea542367719278e0e87b" Jul 28 22:36:50.682776 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:50.682762 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbfb822e360ced9bf6d1482d686464cd7bb498429449ea542367719278e0e87b"} err="failed to get container status \"dbfb822e360ced9bf6d1482d686464cd7bb498429449ea542367719278e0e87b\": rpc error: code = NotFound desc = could not find container \"dbfb822e360ced9bf6d1482d686464cd7bb498429449ea542367719278e0e87b\": container with ID starting with dbfb822e360ced9bf6d1482d686464cd7bb498429449ea542367719278e0e87b not found: ID does not exist" Jul 28 22:36:50.684458 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:50.684439 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-c67c1cbd-openshift-default-f89f647db-6lhvp"] Jul 28 22:36:52.405718 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:36:52.405669 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7" path="/var/lib/kubelet/pods/2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7/volumes" Jul 28 22:37:03.560975 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:03.552279 2553 patch_prober.go:28] interesting pod/e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": read tcp 10.132.0.2:58494->10.132.0.48:15021: read: connection reset by peer" start-of-body= Jul 28 22:37:03.560975 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:03.552352 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c" podUID="abfdf350-2744-4709-989d-9943916f2405" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": read tcp 10.132.0.2:58494->10.132.0.48:15021: read: connection reset by peer" Jul 28 22:37:03.731799 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:03.731764 2553 generic.go:356] "Generic (PLEG): container finished" podID="abfdf350-2744-4709-989d-9943916f2405" containerID="c394744f04bf303be6102d14adec3c91e1b7b9dc5dd4b8dcde957599114e026f" exitCode=0 Jul 28 22:37:03.731976 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:03.731843 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c" event={"ID":"abfdf350-2744-4709-989d-9943916f2405","Type":"ContainerDied","Data":"c394744f04bf303be6102d14adec3c91e1b7b9dc5dd4b8dcde957599114e026f"} Jul 28 22:37:03.732247 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:03.732234 2553 scope.go:117] "RemoveContainer" containerID="c394744f04bf303be6102d14adec3c91e1b7b9dc5dd4b8dcde957599114e026f" Jul 28 22:37:03.733265 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:03.733239 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:37:03.733361 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:03.733320 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:37:03.733423 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:03.733369 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:37:04.740617 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:04.740582 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c" event={"ID":"abfdf350-2744-4709-989d-9943916f2405","Type":"ContainerStarted","Data":"2dd6ed2108a59a4f7e4043b4e121f08ed5a93b45e7a7afeb444472684313ed59"} Jul 28 22:37:05.344086 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:05.344049 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c" Jul 28 22:37:05.344320 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:05.344296 2553 patch_prober.go:28] interesting pod/e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Jul 28 22:37:05.344380 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:05.344350 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c" podUID="abfdf350-2744-4709-989d-9943916f2405" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Jul 28 22:37:06.344189 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:06.344152 2553 patch_prober.go:28] interesting pod/e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Jul 28 22:37:06.344587 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:06.344212 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c" podUID="abfdf350-2744-4709-989d-9943916f2405" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Jul 28 22:37:07.344138 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:07.344101 2553 patch_prober.go:28] interesting pod/e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Jul 28 22:37:07.344324 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:07.344155 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c" podUID="abfdf350-2744-4709-989d-9943916f2405" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Jul 28 22:37:08.344291 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:08.344256 2553 patch_prober.go:28] interesting pod/e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Jul 28 22:37:08.344481 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:08.344306 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c" podUID="abfdf350-2744-4709-989d-9943916f2405" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Jul 28 22:37:09.347572 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:09.347543 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c" Jul 28 22:37:09.347973 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:09.347865 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c" Jul 28 22:37:09.348481 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:09.348466 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c" Jul 28 22:37:13.116399 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:13.116363 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6ad9588a-kjt2x"] Jul 28 22:37:13.116897 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:13.116879 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7" containerName="istio-proxy" Jul 28 22:37:13.116946 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:13.116901 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7" containerName="istio-proxy" Jul 28 22:37:13.116946 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:13.116920 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7" containerName="istio-proxy" Jul 28 22:37:13.116946 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:13.116930 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7" containerName="istio-proxy" Jul 28 22:37:13.117065 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:13.117025 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7" containerName="istio-proxy" Jul 28 22:37:13.117065 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:13.117034 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dca4ce7-8b09-4a9b-9c17-4026f95cb0f7" containerName="istio-proxy" Jul 28 22:37:13.120371 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:13.120346 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6ad9588a-kjt2x" Jul 28 22:37:13.122696 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:13.122671 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qf46x\"" Jul 28 22:37:13.129120 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:13.129037 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6ad9588a-kjt2x"] Jul 28 22:37:13.277780 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:13.277739 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e81ef1dd-2ec9-4250-b63b-76fe4b1a04a1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-6ad9588a-kjt2x\" (UID: \"e81ef1dd-2ec9-4250-b63b-76fe4b1a04a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6ad9588a-kjt2x" Jul 28 22:37:13.379350 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:13.379245 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e81ef1dd-2ec9-4250-b63b-76fe4b1a04a1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-6ad9588a-kjt2x\" (UID: \"e81ef1dd-2ec9-4250-b63b-76fe4b1a04a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6ad9588a-kjt2x" Jul 28 22:37:13.379976 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:13.379955 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e81ef1dd-2ec9-4250-b63b-76fe4b1a04a1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-6ad9588a-kjt2x\" (UID: \"e81ef1dd-2ec9-4250-b63b-76fe4b1a04a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6ad9588a-kjt2x" Jul 28 22:37:13.433795 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:13.433762 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6ad9588a-kjt2x" Jul 28 22:37:13.574044 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:13.574013 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6ad9588a-kjt2x"] Jul 28 22:37:13.576637 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:37:13.576604 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode81ef1dd_2ec9_4250_b63b_76fe4b1a04a1.slice/crio-f19b455af857dd6e3055cbdccd3692dae6b50407118056d85aa62411841f10bf WatchSource:0}: Error finding container f19b455af857dd6e3055cbdccd3692dae6b50407118056d85aa62411841f10bf: Status 404 returned error can't find the container with id f19b455af857dd6e3055cbdccd3692dae6b50407118056d85aa62411841f10bf Jul 28 22:37:13.789037 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:13.788997 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6ad9588a-kjt2x" event={"ID":"e81ef1dd-2ec9-4250-b63b-76fe4b1a04a1","Type":"ContainerStarted","Data":"f19b455af857dd6e3055cbdccd3692dae6b50407118056d85aa62411841f10bf"} Jul 28 22:37:16.809260 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:16.809226 2553 generic.go:356] "Generic (PLEG): container finished" podID="e81ef1dd-2ec9-4250-b63b-76fe4b1a04a1" containerID="4faa3cdcaa7a88bb60e9ddbf8b4a1d7af673690cbde55263ff2907109362c8c4" exitCode=0 Jul 28 22:37:16.809740 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:16.809317 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6ad9588a-kjt2x" event={"ID":"e81ef1dd-2ec9-4250-b63b-76fe4b1a04a1","Type":"ContainerDied","Data":"4faa3cdcaa7a88bb60e9ddbf8b4a1d7af673690cbde55263ff2907109362c8c4"} Jul 28 22:37:17.955871 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:17.955847 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6ad9588a-kjt2x" Jul 28 22:37:18.119147 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:18.119058 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e81ef1dd-2ec9-4250-b63b-76fe4b1a04a1-maas-api-service-ca\") pod \"e81ef1dd-2ec9-4250-b63b-76fe4b1a04a1\" (UID: \"e81ef1dd-2ec9-4250-b63b-76fe4b1a04a1\") " Jul 28 22:37:18.119443 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:18.119421 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e81ef1dd-2ec9-4250-b63b-76fe4b1a04a1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e81ef1dd-2ec9-4250-b63b-76fe4b1a04a1" (UID: "e81ef1dd-2ec9-4250-b63b-76fe4b1a04a1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 22:37:18.220116 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:18.220068 2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e81ef1dd-2ec9-4250-b63b-76fe4b1a04a1-maas-api-service-ca\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:37:18.821532 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:18.821503 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6ad9588a-kjt2x" Jul 28 22:37:18.821733 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:18.821533 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6ad9588a-kjt2x" event={"ID":"e81ef1dd-2ec9-4250-b63b-76fe4b1a04a1","Type":"ContainerDied","Data":"f19b455af857dd6e3055cbdccd3692dae6b50407118056d85aa62411841f10bf"} Jul 28 22:37:18.821733 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:18.821580 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f19b455af857dd6e3055cbdccd3692dae6b50407118056d85aa62411841f10bf" Jul 28 22:37:29.045753 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:29.045694 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6ad9588a-kjt2x"] Jul 28 22:37:29.047747 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:29.047727 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6ad9588a-kjt2x"] Jul 28 22:37:29.192758 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:29.192683 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c"] Jul 28 22:37:29.193058 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:29.192998 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c" podUID="abfdf350-2744-4709-989d-9943916f2405" containerName="istio-proxy" containerID="cri-o://2dd6ed2108a59a4f7e4043b4e121f08ed5a93b45e7a7afeb444472684313ed59" gracePeriod=30 Jul 28 22:37:29.973070 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:29.973033 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7"] Jul 28 22:37:29.973427 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:29.973415 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e81ef1dd-2ec9-4250-b63b-76fe4b1a04a1" containerName="revoke-keys" Jul 28 22:37:29.973469 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:29.973429 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="e81ef1dd-2ec9-4250-b63b-76fe4b1a04a1" containerName="revoke-keys" Jul 28 22:37:29.973517 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:29.973508 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="e81ef1dd-2ec9-4250-b63b-76fe4b1a04a1" containerName="revoke-keys" Jul 28 22:37:29.976741 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:29.976692 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7" Jul 28 22:37:29.979157 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:29.979136 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-ae3227ae-openshift-default-dockercfg-dqbwn\"" Jul 28 22:37:29.989030 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:29.989002 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7"] Jul 28 22:37:30.033947 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:30.033914 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f88bk\" (UniqueName: \"kubernetes.io/projected/2e293242-3fc2-4c3b-9565-cf572c033bb7-kube-api-access-f88bk\") pod \"e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7\" (UID: \"2e293242-3fc2-4c3b-9565-cf572c033bb7\") " pod="openshift-ingress/e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7" Jul 28 22:37:30.034115 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:30.033956 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e293242-3fc2-4c3b-9565-cf572c033bb7-istio-token\") pod \"e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7\" (UID: \"2e293242-3fc2-4c3b-9565-cf572c033bb7\") " pod="openshift-ingress/e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7" Jul 28 22:37:30.034115 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:30.033981 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e293242-3fc2-4c3b-9565-cf572c033bb7-workload-socket\") pod \"e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7\" (UID: \"2e293242-3fc2-4c3b-9565-cf572c033bb7\") " pod="openshift-ingress/e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7" Jul 28 22:37:30.034115 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:30.034025 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e293242-3fc2-4c3b-9565-cf572c033bb7-istio-envoy\") pod \"e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7\" (UID: \"2e293242-3fc2-4c3b-9565-cf572c033bb7\") " pod="openshift-ingress/e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7" Jul 28 22:37:30.034115 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:30.034045 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e293242-3fc2-4c3b-9565-cf572c033bb7-istio-podinfo\") pod \"e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7\" (UID: \"2e293242-3fc2-4c3b-9565-cf572c033bb7\") " pod="openshift-ingress/e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7" Jul 28 22:37:30.034115 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:30.034093 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e293242-3fc2-4c3b-9565-cf572c033bb7-credential-socket\") pod \"e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7\" (UID: \"2e293242-3fc2-4c3b-9565-cf572c033bb7\") " pod="openshift-ingress/e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7" Jul 28 22:37:30.034287 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:30.034129 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e293242-3fc2-4c3b-9565-cf572c033bb7-istio-data\") pod \"e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7\" (UID: \"2e293242-3fc2-4c3b-9565-cf572c033bb7\") " pod="openshift-ingress/e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7" Jul 28 22:37:30.034287 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:30.034163 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e293242-3fc2-4c3b-9565-cf572c033bb7-istiod-ca-cert\") pod \"e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7\" (UID: \"2e293242-3fc2-4c3b-9565-cf572c033bb7\") " pod="openshift-ingress/e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7" Jul 28 22:37:30.034287 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:30.034181 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e293242-3fc2-4c3b-9565-cf572c033bb7-workload-certs\") pod \"e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7\" (UID: \"2e293242-3fc2-4c3b-9565-cf572c033bb7\") " pod="openshift-ingress/e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7" Jul 28 22:37:30.135006 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:30.134967 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e293242-3fc2-4c3b-9565-cf572c033bb7-istio-data\") pod \"e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7\" (UID: \"2e293242-3fc2-4c3b-9565-cf572c033bb7\") " pod="openshift-ingress/e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7" Jul 28 22:37:30.135494 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:30.135031 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e293242-3fc2-4c3b-9565-cf572c033bb7-istiod-ca-cert\") pod \"e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7\" (UID: \"2e293242-3fc2-4c3b-9565-cf572c033bb7\") " pod="openshift-ingress/e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7" Jul 28 22:37:30.135494 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:30.135058 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e293242-3fc2-4c3b-9565-cf572c033bb7-workload-certs\") pod \"e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7\" (UID: \"2e293242-3fc2-4c3b-9565-cf572c033bb7\") " pod="openshift-ingress/e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7" Jul 28 22:37:30.135494 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:30.135121 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f88bk\" (UniqueName: \"kubernetes.io/projected/2e293242-3fc2-4c3b-9565-cf572c033bb7-kube-api-access-f88bk\") pod \"e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7\" (UID: \"2e293242-3fc2-4c3b-9565-cf572c033bb7\") " pod="openshift-ingress/e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7" Jul 28 22:37:30.135494 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:30.135160 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e293242-3fc2-4c3b-9565-cf572c033bb7-istio-token\") pod \"e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7\" (UID: \"2e293242-3fc2-4c3b-9565-cf572c033bb7\") " pod="openshift-ingress/e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7" Jul 28 22:37:30.135494 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:30.135190 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e293242-3fc2-4c3b-9565-cf572c033bb7-workload-socket\") pod \"e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7\" (UID: \"2e293242-3fc2-4c3b-9565-cf572c033bb7\") " pod="openshift-ingress/e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7" Jul 28 22:37:30.135494 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:30.135236 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e293242-3fc2-4c3b-9565-cf572c033bb7-istio-envoy\") pod \"e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7\" (UID: \"2e293242-3fc2-4c3b-9565-cf572c033bb7\") " pod="openshift-ingress/e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7" Jul 28 22:37:30.135494 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:30.135265 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e293242-3fc2-4c3b-9565-cf572c033bb7-istio-podinfo\") pod \"e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7\" (UID: \"2e293242-3fc2-4c3b-9565-cf572c033bb7\") " pod="openshift-ingress/e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7" Jul 28 22:37:30.135494 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:30.135296 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e293242-3fc2-4c3b-9565-cf572c033bb7-credential-socket\") pod \"e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7\" (UID: \"2e293242-3fc2-4c3b-9565-cf572c033bb7\") " pod="openshift-ingress/e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7" Jul 28 22:37:30.135494 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:30.135435 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e293242-3fc2-4c3b-9565-cf572c033bb7-istio-data\") pod \"e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7\" (UID: \"2e293242-3fc2-4c3b-9565-cf572c033bb7\") " pod="openshift-ingress/e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7" Jul 28 22:37:30.136015 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:30.135818 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e293242-3fc2-4c3b-9565-cf572c033bb7-istiod-ca-cert\") pod \"e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7\" (UID: \"2e293242-3fc2-4c3b-9565-cf572c033bb7\") " pod="openshift-ingress/e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7" Jul 28 22:37:30.136153 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:30.136127 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e293242-3fc2-4c3b-9565-cf572c033bb7-workload-certs\") pod \"e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7\" (UID: \"2e293242-3fc2-4c3b-9565-cf572c033bb7\") " pod="openshift-ingress/e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7" Jul 28 22:37:30.136338 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:30.136156 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e293242-3fc2-4c3b-9565-cf572c033bb7-workload-socket\") pod \"e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7\" (UID: \"2e293242-3fc2-4c3b-9565-cf572c033bb7\") " pod="openshift-ingress/e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7" Jul 28 22:37:30.136511 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:30.136483 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e293242-3fc2-4c3b-9565-cf572c033bb7-credential-socket\") pod \"e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7\" (UID: \"2e293242-3fc2-4c3b-9565-cf572c033bb7\") " pod="openshift-ingress/e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7" Jul 28 22:37:30.139196 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:30.139166 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e293242-3fc2-4c3b-9565-cf572c033bb7-istio-podinfo\") pod \"e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7\" (UID: \"2e293242-3fc2-4c3b-9565-cf572c033bb7\") " pod="openshift-ingress/e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7" Jul 28 22:37:30.139418 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:30.139388 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e293242-3fc2-4c3b-9565-cf572c033bb7-istio-envoy\") pod \"e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7\" (UID: \"2e293242-3fc2-4c3b-9565-cf572c033bb7\") " pod="openshift-ingress/e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7" Jul 28 22:37:30.152432 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:30.152402 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e293242-3fc2-4c3b-9565-cf572c033bb7-istio-token\") pod \"e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7\" (UID: \"2e293242-3fc2-4c3b-9565-cf572c033bb7\") " pod="openshift-ingress/e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7" Jul 28 22:37:30.154284 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:30.154261 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f88bk\" (UniqueName: \"kubernetes.io/projected/2e293242-3fc2-4c3b-9565-cf572c033bb7-kube-api-access-f88bk\") pod \"e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7\" (UID: \"2e293242-3fc2-4c3b-9565-cf572c033bb7\") " pod="openshift-ingress/e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7" Jul 28 22:37:30.287430 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:30.287302 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7" Jul 28 22:37:30.408101 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:30.407940 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e81ef1dd-2ec9-4250-b63b-76fe4b1a04a1" path="/var/lib/kubelet/pods/e81ef1dd-2ec9-4250-b63b-76fe4b1a04a1/volumes" Jul 28 22:37:30.439133 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:30.439111 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7"] Jul 28 22:37:30.441321 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:37:30.441292 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e293242_3fc2_4c3b_9565_cf572c033bb7.slice/crio-51f7a9ce1451acdd7db90642e781feb181253703d61a8d415037bf3892a33f96 WatchSource:0}: Error finding container 51f7a9ce1451acdd7db90642e781feb181253703d61a8d415037bf3892a33f96: Status 404 returned error can't find the container with id 51f7a9ce1451acdd7db90642e781feb181253703d61a8d415037bf3892a33f96 Jul 28 22:37:30.443282 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:30.443252 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:37:30.443374 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:30.443335 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:37:30.443449 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:30.443379 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:37:30.889693 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:30.889659 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7" event={"ID":"2e293242-3fc2-4c3b-9565-cf572c033bb7","Type":"ContainerStarted","Data":"cb06086ee52405c44730082724cffa45f042b38b6ede18776ca95782f6a947a5"} Jul 28 22:37:30.889890 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:30.889728 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7" event={"ID":"2e293242-3fc2-4c3b-9565-cf572c033bb7","Type":"ContainerStarted","Data":"51f7a9ce1451acdd7db90642e781feb181253703d61a8d415037bf3892a33f96"} Jul 28 22:37:30.911421 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:30.911367 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7" podStartSLOduration=1.9113514010000001 podStartE2EDuration="1.911351401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:37:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:37:30.909375952 +0000 UTC m=+1670.996634021" watchObservedRunningTime="2026-07-28 22:37:30.911351401 +0000 UTC m=+1670.998609471" Jul 28 22:37:31.287698 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:31.287664 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7" Jul 28 22:37:31.292678 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:31.292648 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7" Jul 28 22:37:31.895231 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:31.895202 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7" Jul 28 22:37:31.896313 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:31.896297 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7" Jul 28 22:37:34.451537 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:34.451508 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c" Jul 28 22:37:34.471282 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:34.471250 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abfdf350-2744-4709-989d-9943916f2405-credential-socket\") pod \"abfdf350-2744-4709-989d-9943916f2405\" (UID: \"abfdf350-2744-4709-989d-9943916f2405\") " Jul 28 22:37:34.471435 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:34.471295 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abfdf350-2744-4709-989d-9943916f2405-istiod-ca-cert\") pod \"abfdf350-2744-4709-989d-9943916f2405\" (UID: \"abfdf350-2744-4709-989d-9943916f2405\") " Jul 28 22:37:34.471435 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:34.471330 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abfdf350-2744-4709-989d-9943916f2405-istio-data\") pod \"abfdf350-2744-4709-989d-9943916f2405\" (UID: \"abfdf350-2744-4709-989d-9943916f2405\") " Jul 28 22:37:34.471435 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:34.471384 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abfdf350-2744-4709-989d-9943916f2405-workload-certs\") pod \"abfdf350-2744-4709-989d-9943916f2405\" (UID: \"abfdf350-2744-4709-989d-9943916f2405\") " Jul 28 22:37:34.471435 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:34.471408 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kw2vj\" (UniqueName: \"kubernetes.io/projected/abfdf350-2744-4709-989d-9943916f2405-kube-api-access-kw2vj\") pod \"abfdf350-2744-4709-989d-9943916f2405\" (UID: \"abfdf350-2744-4709-989d-9943916f2405\") " Jul 28 22:37:34.471655 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:34.471501 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abfdf350-2744-4709-989d-9943916f2405-istio-podinfo\") pod \"abfdf350-2744-4709-989d-9943916f2405\" (UID: \"abfdf350-2744-4709-989d-9943916f2405\") " Jul 28 22:37:34.471655 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:34.471526 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abfdf350-2744-4709-989d-9943916f2405-istio-token\") pod \"abfdf350-2744-4709-989d-9943916f2405\" (UID: \"abfdf350-2744-4709-989d-9943916f2405\") " Jul 28 22:37:34.471655 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:34.471536 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abfdf350-2744-4709-989d-9943916f2405-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "abfdf350-2744-4709-989d-9943916f2405" (UID: "abfdf350-2744-4709-989d-9943916f2405"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:37:34.471655 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:34.471558 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abfdf350-2744-4709-989d-9943916f2405-istio-envoy\") pod \"abfdf350-2744-4709-989d-9943916f2405\" (UID: \"abfdf350-2744-4709-989d-9943916f2405\") " Jul 28 22:37:34.471655 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:34.471582 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abfdf350-2744-4709-989d-9943916f2405-workload-socket\") pod \"abfdf350-2744-4709-989d-9943916f2405\" (UID: \"abfdf350-2744-4709-989d-9943916f2405\") " Jul 28 22:37:34.471946 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:34.471774 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abfdf350-2744-4709-989d-9943916f2405-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "abfdf350-2744-4709-989d-9943916f2405" (UID: "abfdf350-2744-4709-989d-9943916f2405"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:37:34.471946 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:34.471796 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abfdf350-2744-4709-989d-9943916f2405-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "abfdf350-2744-4709-989d-9943916f2405" (UID: "abfdf350-2744-4709-989d-9943916f2405"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 22:37:34.472033 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:34.471976 2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abfdf350-2744-4709-989d-9943916f2405-credential-socket\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:37:34.472033 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:34.471992 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abfdf350-2744-4709-989d-9943916f2405-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "abfdf350-2744-4709-989d-9943916f2405" (UID: "abfdf350-2744-4709-989d-9943916f2405"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:37:34.472033 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:34.471999 2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abfdf350-2744-4709-989d-9943916f2405-istiod-ca-cert\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:37:34.472033 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:34.472030 2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abfdf350-2744-4709-989d-9943916f2405-workload-certs\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:37:34.472347 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:34.472321 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abfdf350-2744-4709-989d-9943916f2405-istio-data" (OuterVolumeSpecName: "istio-data") pod "abfdf350-2744-4709-989d-9943916f2405" (UID: "abfdf350-2744-4709-989d-9943916f2405"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:37:34.473817 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:34.473797 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abfdf350-2744-4709-989d-9943916f2405-istio-token" (OuterVolumeSpecName: "istio-token") pod "abfdf350-2744-4709-989d-9943916f2405" (UID: "abfdf350-2744-4709-989d-9943916f2405"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:37:34.474504 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:34.474480 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abfdf350-2744-4709-989d-9943916f2405-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "abfdf350-2744-4709-989d-9943916f2405" (UID: "abfdf350-2744-4709-989d-9943916f2405"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:37:34.474583 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:34.474539 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/abfdf350-2744-4709-989d-9943916f2405-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "abfdf350-2744-4709-989d-9943916f2405" (UID: "abfdf350-2744-4709-989d-9943916f2405"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 22:37:34.474583 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:34.474544 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abfdf350-2744-4709-989d-9943916f2405-kube-api-access-kw2vj" (OuterVolumeSpecName: "kube-api-access-kw2vj") pod "abfdf350-2744-4709-989d-9943916f2405" (UID: "abfdf350-2744-4709-989d-9943916f2405"). InnerVolumeSpecName "kube-api-access-kw2vj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:37:34.572417 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:34.572385 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kw2vj\" (UniqueName: \"kubernetes.io/projected/abfdf350-2744-4709-989d-9943916f2405-kube-api-access-kw2vj\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:37:34.572417 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:34.572410 2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abfdf350-2744-4709-989d-9943916f2405-istio-podinfo\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:37:34.572417 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:34.572419 2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abfdf350-2744-4709-989d-9943916f2405-istio-token\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:37:34.572417 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:34.572427 2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abfdf350-2744-4709-989d-9943916f2405-istio-envoy\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:37:34.572759 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:34.572435 2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abfdf350-2744-4709-989d-9943916f2405-workload-socket\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:37:34.572759 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:34.572443 2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abfdf350-2744-4709-989d-9943916f2405-istio-data\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:37:34.912218 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:34.912124 2553 generic.go:356] "Generic (PLEG): container finished" podID="abfdf350-2744-4709-989d-9943916f2405" containerID="2dd6ed2108a59a4f7e4043b4e121f08ed5a93b45e7a7afeb444472684313ed59" exitCode=0 Jul 28 22:37:34.912218 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:34.912176 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c" Jul 28 22:37:34.912447 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:34.912211 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c" event={"ID":"abfdf350-2744-4709-989d-9943916f2405","Type":"ContainerDied","Data":"2dd6ed2108a59a4f7e4043b4e121f08ed5a93b45e7a7afeb444472684313ed59"} Jul 28 22:37:34.912447 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:34.912272 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c" event={"ID":"abfdf350-2744-4709-989d-9943916f2405","Type":"ContainerDied","Data":"471edf0f934a7e157c1d2281336ade87baacd03fdd660c36ebb0b04930bb740a"} Jul 28 22:37:34.912447 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:34.912305 2553 scope.go:117] "RemoveContainer" containerID="2dd6ed2108a59a4f7e4043b4e121f08ed5a93b45e7a7afeb444472684313ed59" Jul 28 22:37:34.921733 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:34.921698 2553 scope.go:117] "RemoveContainer" containerID="c394744f04bf303be6102d14adec3c91e1b7b9dc5dd4b8dcde957599114e026f" Jul 28 22:37:34.930111 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:34.930088 2553 scope.go:117] "RemoveContainer" containerID="2dd6ed2108a59a4f7e4043b4e121f08ed5a93b45e7a7afeb444472684313ed59" Jul 28 22:37:34.930459 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:37:34.930430 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dd6ed2108a59a4f7e4043b4e121f08ed5a93b45e7a7afeb444472684313ed59\": container with ID starting with 2dd6ed2108a59a4f7e4043b4e121f08ed5a93b45e7a7afeb444472684313ed59 not found: ID does not exist" containerID="2dd6ed2108a59a4f7e4043b4e121f08ed5a93b45e7a7afeb444472684313ed59" Jul 28 22:37:34.930593 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:34.930470 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dd6ed2108a59a4f7e4043b4e121f08ed5a93b45e7a7afeb444472684313ed59"} err="failed to get container status \"2dd6ed2108a59a4f7e4043b4e121f08ed5a93b45e7a7afeb444472684313ed59\": rpc error: code = NotFound desc = could not find container \"2dd6ed2108a59a4f7e4043b4e121f08ed5a93b45e7a7afeb444472684313ed59\": container with ID starting with 2dd6ed2108a59a4f7e4043b4e121f08ed5a93b45e7a7afeb444472684313ed59 not found: ID does not exist" Jul 28 22:37:34.930593 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:34.930491 2553 scope.go:117] "RemoveContainer" containerID="c394744f04bf303be6102d14adec3c91e1b7b9dc5dd4b8dcde957599114e026f" Jul 28 22:37:34.930834 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:37:34.930816 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c394744f04bf303be6102d14adec3c91e1b7b9dc5dd4b8dcde957599114e026f\": container with ID starting with c394744f04bf303be6102d14adec3c91e1b7b9dc5dd4b8dcde957599114e026f not found: ID does not exist" containerID="c394744f04bf303be6102d14adec3c91e1b7b9dc5dd4b8dcde957599114e026f" Jul 28 22:37:34.930882 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:34.930842 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c394744f04bf303be6102d14adec3c91e1b7b9dc5dd4b8dcde957599114e026f"} err="failed to get container status \"c394744f04bf303be6102d14adec3c91e1b7b9dc5dd4b8dcde957599114e026f\": rpc error: code = NotFound desc = could not find container \"c394744f04bf303be6102d14adec3c91e1b7b9dc5dd4b8dcde957599114e026f\": container with ID starting with c394744f04bf303be6102d14adec3c91e1b7b9dc5dd4b8dcde957599114e026f not found: ID does not exist" Jul 28 22:37:34.932309 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:34.932292 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c"] Jul 28 22:37:34.939436 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:34.939414 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-6ad9588a-gw-openshift-default-6d79d644f6-cgh8c"] Jul 28 22:37:36.405201 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:36.405167 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abfdf350-2744-4709-989d-9943916f2405" path="/var/lib/kubelet/pods/abfdf350-2744-4709-989d-9943916f2405/volumes" Jul 28 22:37:47.612860 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:47.612019 2553 patch_prober.go:28] interesting pod/e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": read tcp 10.132.0.2:48084->10.132.0.50:15021: read: connection reset by peer" start-of-body= Jul 28 22:37:47.612860 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:47.612095 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7" podUID="2e293242-3fc2-4c3b-9565-cf572c033bb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": read tcp 10.132.0.2:48084->10.132.0.50:15021: read: connection reset by peer" Jul 28 22:37:47.983043 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:47.983006 2553 generic.go:356] "Generic (PLEG): container finished" podID="2e293242-3fc2-4c3b-9565-cf572c033bb7" containerID="cb06086ee52405c44730082724cffa45f042b38b6ede18776ca95782f6a947a5" exitCode=0 Jul 28 22:37:47.983237 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:47.983078 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7" event={"ID":"2e293242-3fc2-4c3b-9565-cf572c033bb7","Type":"ContainerDied","Data":"cb06086ee52405c44730082724cffa45f042b38b6ede18776ca95782f6a947a5"} Jul 28 22:37:47.983519 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:47.983501 2553 scope.go:117] "RemoveContainer" containerID="cb06086ee52405c44730082724cffa45f042b38b6ede18776ca95782f6a947a5" Jul 28 22:37:47.984563 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:47.984534 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:37:47.984623 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:47.984607 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:37:47.984659 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:47.984649 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:37:48.990126 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:48.990089 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7" event={"ID":"2e293242-3fc2-4c3b-9565-cf572c033bb7","Type":"ContainerStarted","Data":"3e8308e02501ae416aadee550c7a9c93d8647e0fb4a96a30374e352fdf4717ab"} Jul 28 22:37:49.288418 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:49.288331 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7" Jul 28 22:37:49.288608 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:49.288587 2553 patch_prober.go:28] interesting pod/e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Jul 28 22:37:49.288683 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:49.288645 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7" podUID="2e293242-3fc2-4c3b-9565-cf572c033bb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Jul 28 22:37:50.288651 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:50.288614 2553 patch_prober.go:28] interesting pod/e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Jul 28 22:37:50.289060 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:50.288680 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7" podUID="2e293242-3fc2-4c3b-9565-cf572c033bb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Jul 28 22:37:51.288128 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:51.288090 2553 patch_prober.go:28] interesting pod/e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Jul 28 22:37:51.288309 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:51.288148 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7" podUID="2e293242-3fc2-4c3b-9565-cf572c033bb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Jul 28 22:37:52.287909 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:52.287874 2553 patch_prober.go:28] interesting pod/e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Jul 28 22:37:52.288327 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:52.287932 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7" podUID="2e293242-3fc2-4c3b-9565-cf572c033bb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Jul 28 22:37:53.292035 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:53.292007 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7" Jul 28 22:37:53.292522 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:53.292505 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7" Jul 28 22:37:53.293229 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:37:53.293210 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7" Jul 28 22:38:04.344497 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:04.344461 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7"] Jul 28 22:38:04.345109 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:04.344670 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7" podUID="2e293242-3fc2-4c3b-9565-cf572c033bb7" containerName="istio-proxy" containerID="cri-o://3e8308e02501ae416aadee550c7a9c93d8647e0fb4a96a30374e352fdf4717ab" gracePeriod=30 Jul 28 22:38:05.125011 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:05.124980 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml"] Jul 28 22:38:05.125366 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:05.125354 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abfdf350-2744-4709-989d-9943916f2405" containerName="istio-proxy" Jul 28 22:38:05.125414 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:05.125368 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="abfdf350-2744-4709-989d-9943916f2405" containerName="istio-proxy" Jul 28 22:38:05.125414 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:05.125380 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abfdf350-2744-4709-989d-9943916f2405" containerName="istio-proxy" Jul 28 22:38:05.125414 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:05.125385 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="abfdf350-2744-4709-989d-9943916f2405" containerName="istio-proxy" Jul 28 22:38:05.125511 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:05.125452 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="abfdf350-2744-4709-989d-9943916f2405" containerName="istio-proxy" Jul 28 22:38:05.125511 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:05.125461 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="abfdf350-2744-4709-989d-9943916f2405" containerName="istio-proxy" Jul 28 22:38:05.131853 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:05.131831 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml" Jul 28 22:38:05.134022 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:05.134003 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-b9e632fc-openshift-default-dockercfg-xklhs\"" Jul 28 22:38:05.140352 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:05.140328 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml"] Jul 28 22:38:05.259553 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:05.259518 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b4538f7-19c8-48cb-aebb-3f9666f9347f-istio-token\") pod \"e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml\" (UID: \"9b4538f7-19c8-48cb-aebb-3f9666f9347f\") " pod="openshift-ingress/e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml" Jul 28 22:38:05.259553 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:05.259557 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgtc6\" (UniqueName: \"kubernetes.io/projected/9b4538f7-19c8-48cb-aebb-3f9666f9347f-kube-api-access-mgtc6\") pod \"e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml\" (UID: \"9b4538f7-19c8-48cb-aebb-3f9666f9347f\") " pod="openshift-ingress/e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml" Jul 28 22:38:05.259812 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:05.259589 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b4538f7-19c8-48cb-aebb-3f9666f9347f-credential-socket\") pod \"e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml\" (UID: \"9b4538f7-19c8-48cb-aebb-3f9666f9347f\") " pod="openshift-ingress/e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml" Jul 28 22:38:05.259812 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:05.259621 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b4538f7-19c8-48cb-aebb-3f9666f9347f-istio-data\") pod \"e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml\" (UID: \"9b4538f7-19c8-48cb-aebb-3f9666f9347f\") " pod="openshift-ingress/e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml" Jul 28 22:38:05.259812 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:05.259667 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b4538f7-19c8-48cb-aebb-3f9666f9347f-istio-envoy\") pod \"e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml\" (UID: \"9b4538f7-19c8-48cb-aebb-3f9666f9347f\") " pod="openshift-ingress/e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml" Jul 28 22:38:05.259812 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:05.259748 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b4538f7-19c8-48cb-aebb-3f9666f9347f-istiod-ca-cert\") pod \"e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml\" (UID: \"9b4538f7-19c8-48cb-aebb-3f9666f9347f\") " pod="openshift-ingress/e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml" Jul 28 22:38:05.259812 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:05.259778 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b4538f7-19c8-48cb-aebb-3f9666f9347f-workload-certs\") pod \"e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml\" (UID: \"9b4538f7-19c8-48cb-aebb-3f9666f9347f\") " pod="openshift-ingress/e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml" Jul 28 22:38:05.259812 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:05.259797 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b4538f7-19c8-48cb-aebb-3f9666f9347f-istio-podinfo\") pod \"e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml\" (UID: \"9b4538f7-19c8-48cb-aebb-3f9666f9347f\") " pod="openshift-ingress/e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml" Jul 28 22:38:05.260004 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:05.259868 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b4538f7-19c8-48cb-aebb-3f9666f9347f-workload-socket\") pod \"e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml\" (UID: \"9b4538f7-19c8-48cb-aebb-3f9666f9347f\") " pod="openshift-ingress/e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml" Jul 28 22:38:05.360342 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:05.360308 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b4538f7-19c8-48cb-aebb-3f9666f9347f-istiod-ca-cert\") pod \"e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml\" (UID: \"9b4538f7-19c8-48cb-aebb-3f9666f9347f\") " pod="openshift-ingress/e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml" Jul 28 22:38:05.360873 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:05.360361 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b4538f7-19c8-48cb-aebb-3f9666f9347f-workload-certs\") pod \"e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml\" (UID: \"9b4538f7-19c8-48cb-aebb-3f9666f9347f\") " pod="openshift-ingress/e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml" Jul 28 22:38:05.360873 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:05.360483 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b4538f7-19c8-48cb-aebb-3f9666f9347f-istio-podinfo\") pod \"e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml\" (UID: \"9b4538f7-19c8-48cb-aebb-3f9666f9347f\") " pod="openshift-ingress/e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml" Jul 28 22:38:05.360873 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:05.360540 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b4538f7-19c8-48cb-aebb-3f9666f9347f-workload-socket\") pod \"e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml\" (UID: \"9b4538f7-19c8-48cb-aebb-3f9666f9347f\") " pod="openshift-ingress/e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml" Jul 28 22:38:05.360873 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:05.360566 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b4538f7-19c8-48cb-aebb-3f9666f9347f-istio-token\") pod \"e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml\" (UID: \"9b4538f7-19c8-48cb-aebb-3f9666f9347f\") " pod="openshift-ingress/e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml" Jul 28 22:38:05.360873 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:05.360583 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgtc6\" (UniqueName: \"kubernetes.io/projected/9b4538f7-19c8-48cb-aebb-3f9666f9347f-kube-api-access-mgtc6\") pod \"e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml\" (UID: \"9b4538f7-19c8-48cb-aebb-3f9666f9347f\") " pod="openshift-ingress/e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml" Jul 28 22:38:05.360873 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:05.360611 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b4538f7-19c8-48cb-aebb-3f9666f9347f-credential-socket\") pod \"e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml\" (UID: \"9b4538f7-19c8-48cb-aebb-3f9666f9347f\") " pod="openshift-ingress/e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml" Jul 28 22:38:05.360873 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:05.360636 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b4538f7-19c8-48cb-aebb-3f9666f9347f-istio-data\") pod \"e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml\" (UID: \"9b4538f7-19c8-48cb-aebb-3f9666f9347f\") " pod="openshift-ingress/e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml" Jul 28 22:38:05.360873 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:05.360669 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b4538f7-19c8-48cb-aebb-3f9666f9347f-istio-envoy\") pod \"e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml\" (UID: \"9b4538f7-19c8-48cb-aebb-3f9666f9347f\") " pod="openshift-ingress/e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml" Jul 28 22:38:05.360873 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:05.360808 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b4538f7-19c8-48cb-aebb-3f9666f9347f-workload-certs\") pod \"e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml\" (UID: \"9b4538f7-19c8-48cb-aebb-3f9666f9347f\") " pod="openshift-ingress/e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml" Jul 28 22:38:05.361382 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:05.360962 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b4538f7-19c8-48cb-aebb-3f9666f9347f-workload-socket\") pod \"e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml\" (UID: \"9b4538f7-19c8-48cb-aebb-3f9666f9347f\") " pod="openshift-ingress/e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml" Jul 28 22:38:05.361382 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:05.361083 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b4538f7-19c8-48cb-aebb-3f9666f9347f-credential-socket\") pod \"e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml\" (UID: \"9b4538f7-19c8-48cb-aebb-3f9666f9347f\") " pod="openshift-ingress/e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml" Jul 28 22:38:05.361382 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:05.361226 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b4538f7-19c8-48cb-aebb-3f9666f9347f-istio-data\") pod \"e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml\" (UID: \"9b4538f7-19c8-48cb-aebb-3f9666f9347f\") " pod="openshift-ingress/e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml" Jul 28 22:38:05.361382 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:05.361366 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b4538f7-19c8-48cb-aebb-3f9666f9347f-istiod-ca-cert\") pod \"e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml\" (UID: \"9b4538f7-19c8-48cb-aebb-3f9666f9347f\") " pod="openshift-ingress/e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml" Jul 28 22:38:05.363027 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:05.363002 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b4538f7-19c8-48cb-aebb-3f9666f9347f-istio-envoy\") pod \"e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml\" (UID: \"9b4538f7-19c8-48cb-aebb-3f9666f9347f\") " pod="openshift-ingress/e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml" Jul 28 22:38:05.363234 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:05.363218 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b4538f7-19c8-48cb-aebb-3f9666f9347f-istio-podinfo\") pod \"e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml\" (UID: \"9b4538f7-19c8-48cb-aebb-3f9666f9347f\") " pod="openshift-ingress/e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml" Jul 28 22:38:05.369127 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:05.369101 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b4538f7-19c8-48cb-aebb-3f9666f9347f-istio-token\") pod \"e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml\" (UID: \"9b4538f7-19c8-48cb-aebb-3f9666f9347f\") " pod="openshift-ingress/e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml" Jul 28 22:38:05.369254 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:05.369174 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgtc6\" (UniqueName: \"kubernetes.io/projected/9b4538f7-19c8-48cb-aebb-3f9666f9347f-kube-api-access-mgtc6\") pod \"e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml\" (UID: \"9b4538f7-19c8-48cb-aebb-3f9666f9347f\") " pod="openshift-ingress/e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml" Jul 28 22:38:05.443904 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:05.443811 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml" Jul 28 22:38:05.591831 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:05.591804 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml"] Jul 28 22:38:05.593774 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:38:05.593696 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b4538f7_19c8_48cb_aebb_3f9666f9347f.slice/crio-8943e8b5c0648e7f2bd86b20d63e97e85732ffef5d87c5b90e7591db6e86499b WatchSource:0}: Error finding container 8943e8b5c0648e7f2bd86b20d63e97e85732ffef5d87c5b90e7591db6e86499b: Status 404 returned error can't find the container with id 8943e8b5c0648e7f2bd86b20d63e97e85732ffef5d87c5b90e7591db6e86499b Jul 28 22:38:05.595842 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:05.595811 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:38:05.595931 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:05.595920 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:38:05.595992 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:05.595952 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:38:06.084322 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:06.084288 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml" event={"ID":"9b4538f7-19c8-48cb-aebb-3f9666f9347f","Type":"ContainerStarted","Data":"41438027eb2a127149e35d91260d73283464775073304ccf9a962ca6c2a77341"} Jul 28 22:38:06.084504 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:06.084337 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml" event={"ID":"9b4538f7-19c8-48cb-aebb-3f9666f9347f","Type":"ContainerStarted","Data":"8943e8b5c0648e7f2bd86b20d63e97e85732ffef5d87c5b90e7591db6e86499b"} Jul 28 22:38:06.108079 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:06.108029 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml" podStartSLOduration=1.108014771 podStartE2EDuration="1.108014771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:38:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:38:06.106445792 +0000 UTC m=+1706.193703880" watchObservedRunningTime="2026-07-28 22:38:06.108014771 +0000 UTC m=+1706.195272840" Jul 28 22:38:06.444069 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:06.443970 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml" Jul 28 22:38:06.449418 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:06.449394 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml" Jul 28 22:38:07.090099 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:07.090070 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml" Jul 28 22:38:07.091083 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:07.091058 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml" Jul 28 22:38:08.295178 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:08.295142 2553 patch_prober.go:28] interesting pod/e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Jul 28 22:38:08.295584 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:08.295204 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7" podUID="2e293242-3fc2-4c3b-9565-cf572c033bb7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 22:38:09.591459 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:09.591434 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7" Jul 28 22:38:09.707638 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:09.707549 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f88bk\" (UniqueName: \"kubernetes.io/projected/2e293242-3fc2-4c3b-9565-cf572c033bb7-kube-api-access-f88bk\") pod \"2e293242-3fc2-4c3b-9565-cf572c033bb7\" (UID: \"2e293242-3fc2-4c3b-9565-cf572c033bb7\") " Jul 28 22:38:09.707638 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:09.707605 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e293242-3fc2-4c3b-9565-cf572c033bb7-istio-envoy\") pod \"2e293242-3fc2-4c3b-9565-cf572c033bb7\" (UID: \"2e293242-3fc2-4c3b-9565-cf572c033bb7\") " Jul 28 22:38:09.707895 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:09.707669 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e293242-3fc2-4c3b-9565-cf572c033bb7-credential-socket\") pod \"2e293242-3fc2-4c3b-9565-cf572c033bb7\" (UID: \"2e293242-3fc2-4c3b-9565-cf572c033bb7\") " Jul 28 22:38:09.707895 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:09.707736 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e293242-3fc2-4c3b-9565-cf572c033bb7-istio-token\") pod \"2e293242-3fc2-4c3b-9565-cf572c033bb7\" (UID: \"2e293242-3fc2-4c3b-9565-cf572c033bb7\") " Jul 28 22:38:09.707895 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:09.707775 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e293242-3fc2-4c3b-9565-cf572c033bb7-istio-data\") pod \"2e293242-3fc2-4c3b-9565-cf572c033bb7\" (UID: \"2e293242-3fc2-4c3b-9565-cf572c033bb7\") " Jul 28 22:38:09.707895 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:09.707806 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e293242-3fc2-4c3b-9565-cf572c033bb7-workload-socket\") pod \"2e293242-3fc2-4c3b-9565-cf572c033bb7\" (UID: \"2e293242-3fc2-4c3b-9565-cf572c033bb7\") " Jul 28 22:38:09.707895 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:09.707837 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e293242-3fc2-4c3b-9565-cf572c033bb7-workload-certs\") pod \"2e293242-3fc2-4c3b-9565-cf572c033bb7\" (UID: \"2e293242-3fc2-4c3b-9565-cf572c033bb7\") " Jul 28 22:38:09.707895 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:09.707864 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e293242-3fc2-4c3b-9565-cf572c033bb7-istiod-ca-cert\") pod \"2e293242-3fc2-4c3b-9565-cf572c033bb7\" (UID: \"2e293242-3fc2-4c3b-9565-cf572c033bb7\") " Jul 28 22:38:09.707895 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:09.707885 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e293242-3fc2-4c3b-9565-cf572c033bb7-istio-podinfo\") pod \"2e293242-3fc2-4c3b-9565-cf572c033bb7\" (UID: \"2e293242-3fc2-4c3b-9565-cf572c033bb7\") " Jul 28 22:38:09.708269 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:09.708143 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e293242-3fc2-4c3b-9565-cf572c033bb7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2e293242-3fc2-4c3b-9565-cf572c033bb7" (UID: "2e293242-3fc2-4c3b-9565-cf572c033bb7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:38:09.708382 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:09.708295 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e293242-3fc2-4c3b-9565-cf572c033bb7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2e293242-3fc2-4c3b-9565-cf572c033bb7" (UID: "2e293242-3fc2-4c3b-9565-cf572c033bb7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:38:09.708453 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:09.708383 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e293242-3fc2-4c3b-9565-cf572c033bb7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2e293242-3fc2-4c3b-9565-cf572c033bb7" (UID: "2e293242-3fc2-4c3b-9565-cf572c033bb7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:38:09.709052 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:09.709024 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e293242-3fc2-4c3b-9565-cf572c033bb7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2e293242-3fc2-4c3b-9565-cf572c033bb7" (UID: "2e293242-3fc2-4c3b-9565-cf572c033bb7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 22:38:09.709341 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:09.709318 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e293242-3fc2-4c3b-9565-cf572c033bb7-istio-data" (OuterVolumeSpecName: "istio-data") pod "2e293242-3fc2-4c3b-9565-cf572c033bb7" (UID: "2e293242-3fc2-4c3b-9565-cf572c033bb7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:38:09.710298 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:09.710268 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e293242-3fc2-4c3b-9565-cf572c033bb7-istio-token" (OuterVolumeSpecName: "istio-token") pod "2e293242-3fc2-4c3b-9565-cf572c033bb7" (UID: "2e293242-3fc2-4c3b-9565-cf572c033bb7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:38:09.710397 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:09.710330 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e293242-3fc2-4c3b-9565-cf572c033bb7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2e293242-3fc2-4c3b-9565-cf572c033bb7" (UID: "2e293242-3fc2-4c3b-9565-cf572c033bb7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 22:38:09.710504 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:09.710476 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e293242-3fc2-4c3b-9565-cf572c033bb7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2e293242-3fc2-4c3b-9565-cf572c033bb7" (UID: "2e293242-3fc2-4c3b-9565-cf572c033bb7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:38:09.710620 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:09.710546 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e293242-3fc2-4c3b-9565-cf572c033bb7-kube-api-access-f88bk" (OuterVolumeSpecName: "kube-api-access-f88bk") pod "2e293242-3fc2-4c3b-9565-cf572c033bb7" (UID: "2e293242-3fc2-4c3b-9565-cf572c033bb7"). InnerVolumeSpecName "kube-api-access-f88bk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:38:09.808658 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:09.808626 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f88bk\" (UniqueName: \"kubernetes.io/projected/2e293242-3fc2-4c3b-9565-cf572c033bb7-kube-api-access-f88bk\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:38:09.808658 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:09.808655 2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e293242-3fc2-4c3b-9565-cf572c033bb7-istio-envoy\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:38:09.808658 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:09.808664 2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e293242-3fc2-4c3b-9565-cf572c033bb7-credential-socket\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:38:09.808902 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:09.808672 2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e293242-3fc2-4c3b-9565-cf572c033bb7-istio-token\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:38:09.808902 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:09.808681 2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e293242-3fc2-4c3b-9565-cf572c033bb7-istio-data\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:38:09.808902 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:09.808688 2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e293242-3fc2-4c3b-9565-cf572c033bb7-workload-socket\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:38:09.808902 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:09.808696 2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e293242-3fc2-4c3b-9565-cf572c033bb7-workload-certs\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:38:09.808902 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:09.808730 2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e293242-3fc2-4c3b-9565-cf572c033bb7-istiod-ca-cert\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:38:09.808902 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:09.808740 2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e293242-3fc2-4c3b-9565-cf572c033bb7-istio-podinfo\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:38:10.105998 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:10.105964 2553 generic.go:356] "Generic (PLEG): container finished" podID="2e293242-3fc2-4c3b-9565-cf572c033bb7" containerID="3e8308e02501ae416aadee550c7a9c93d8647e0fb4a96a30374e352fdf4717ab" exitCode=0 Jul 28 22:38:10.106169 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:10.106022 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7" Jul 28 22:38:10.106169 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:10.106052 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7" event={"ID":"2e293242-3fc2-4c3b-9565-cf572c033bb7","Type":"ContainerDied","Data":"3e8308e02501ae416aadee550c7a9c93d8647e0fb4a96a30374e352fdf4717ab"} Jul 28 22:38:10.106169 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:10.106140 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7" event={"ID":"2e293242-3fc2-4c3b-9565-cf572c033bb7","Type":"ContainerDied","Data":"51f7a9ce1451acdd7db90642e781feb181253703d61a8d415037bf3892a33f96"} Jul 28 22:38:10.106169 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:10.106168 2553 scope.go:117] "RemoveContainer" containerID="3e8308e02501ae416aadee550c7a9c93d8647e0fb4a96a30374e352fdf4717ab" Jul 28 22:38:10.116319 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:10.116299 2553 scope.go:117] "RemoveContainer" containerID="cb06086ee52405c44730082724cffa45f042b38b6ede18776ca95782f6a947a5" Jul 28 22:38:10.124727 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:10.124691 2553 scope.go:117] "RemoveContainer" containerID="3e8308e02501ae416aadee550c7a9c93d8647e0fb4a96a30374e352fdf4717ab" Jul 28 22:38:10.125036 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:38:10.124981 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e8308e02501ae416aadee550c7a9c93d8647e0fb4a96a30374e352fdf4717ab\": container with ID starting with 3e8308e02501ae416aadee550c7a9c93d8647e0fb4a96a30374e352fdf4717ab not found: ID does not exist" containerID="3e8308e02501ae416aadee550c7a9c93d8647e0fb4a96a30374e352fdf4717ab" Jul 28 22:38:10.125036 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:10.125019 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e8308e02501ae416aadee550c7a9c93d8647e0fb4a96a30374e352fdf4717ab"} err="failed to get container status \"3e8308e02501ae416aadee550c7a9c93d8647e0fb4a96a30374e352fdf4717ab\": rpc error: code = NotFound desc = could not find container \"3e8308e02501ae416aadee550c7a9c93d8647e0fb4a96a30374e352fdf4717ab\": container with ID starting with 3e8308e02501ae416aadee550c7a9c93d8647e0fb4a96a30374e352fdf4717ab not found: ID does not exist" Jul 28 22:38:10.125273 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:10.125044 2553 scope.go:117] "RemoveContainer" containerID="cb06086ee52405c44730082724cffa45f042b38b6ede18776ca95782f6a947a5" Jul 28 22:38:10.125399 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:38:10.125355 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb06086ee52405c44730082724cffa45f042b38b6ede18776ca95782f6a947a5\": container with ID starting with cb06086ee52405c44730082724cffa45f042b38b6ede18776ca95782f6a947a5 not found: ID does not exist" containerID="cb06086ee52405c44730082724cffa45f042b38b6ede18776ca95782f6a947a5" Jul 28 22:38:10.125492 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:10.125392 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb06086ee52405c44730082724cffa45f042b38b6ede18776ca95782f6a947a5"} err="failed to get container status \"cb06086ee52405c44730082724cffa45f042b38b6ede18776ca95782f6a947a5\": rpc error: code = NotFound desc = could not find container \"cb06086ee52405c44730082724cffa45f042b38b6ede18776ca95782f6a947a5\": container with ID starting with cb06086ee52405c44730082724cffa45f042b38b6ede18776ca95782f6a947a5 not found: ID does not exist" Jul 28 22:38:10.128167 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:10.128145 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7"] Jul 28 22:38:10.134047 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:10.134029 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-ae3227ae-openshift-default-7b554db5dd-sbjf7"] Jul 28 22:38:10.405110 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:10.405027 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e293242-3fc2-4c3b-9565-cf572c033bb7" path="/var/lib/kubelet/pods/2e293242-3fc2-4c3b-9565-cf572c033bb7/volumes" Jul 28 22:38:21.174164 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:21.174127 2553 generic.go:356] "Generic (PLEG): container finished" podID="9b4538f7-19c8-48cb-aebb-3f9666f9347f" containerID="41438027eb2a127149e35d91260d73283464775073304ccf9a962ca6c2a77341" exitCode=0 Jul 28 22:38:21.174647 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:21.174182 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml" event={"ID":"9b4538f7-19c8-48cb-aebb-3f9666f9347f","Type":"ContainerDied","Data":"41438027eb2a127149e35d91260d73283464775073304ccf9a962ca6c2a77341"} Jul 28 22:38:21.174647 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:21.174536 2553 scope.go:117] "RemoveContainer" containerID="41438027eb2a127149e35d91260d73283464775073304ccf9a962ca6c2a77341" Jul 28 22:38:21.175549 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:21.175521 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:38:21.175658 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:21.175600 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:38:21.175658 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:21.175646 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:38:22.090229 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:22.090184 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml" Jul 28 22:38:22.181604 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:22.181571 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml" event={"ID":"9b4538f7-19c8-48cb-aebb-3f9666f9347f","Type":"ContainerStarted","Data":"2a7bbdca3305341b238e9fe53f0f15230ccfbe4d35f7a4486eaba2a8e7577470"} Jul 28 22:38:22.444539 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:22.444452 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml" Jul 28 22:38:22.445716 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:22.445675 2553 patch_prober.go:28] interesting pod/e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Jul 28 22:38:22.445835 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:22.445743 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml" podUID="9b4538f7-19c8-48cb-aebb-3f9666f9347f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Jul 28 22:38:23.444313 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:23.444275 2553 patch_prober.go:28] interesting pod/e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Jul 28 22:38:23.444751 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:23.444335 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml" podUID="9b4538f7-19c8-48cb-aebb-3f9666f9347f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Jul 28 22:38:24.444863 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:24.444826 2553 patch_prober.go:28] interesting pod/e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Jul 28 22:38:24.445252 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:24.444886 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml" podUID="9b4538f7-19c8-48cb-aebb-3f9666f9347f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Jul 28 22:38:25.444353 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:25.444313 2553 patch_prober.go:28] interesting pod/e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Jul 28 22:38:25.444532 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:25.444399 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml" podUID="9b4538f7-19c8-48cb-aebb-3f9666f9347f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Jul 28 22:38:26.448382 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:26.448350 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml" Jul 28 22:38:26.448809 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:26.448598 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml" Jul 28 22:38:26.449330 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:26.449316 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml" Jul 28 22:38:32.323551 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:32.323515 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b9e632fc-p24tk"] Jul 28 22:38:32.323984 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:32.323918 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e293242-3fc2-4c3b-9565-cf572c033bb7" containerName="istio-proxy" Jul 28 22:38:32.323984 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:32.323932 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e293242-3fc2-4c3b-9565-cf572c033bb7" containerName="istio-proxy" Jul 28 22:38:32.323984 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:32.323940 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e293242-3fc2-4c3b-9565-cf572c033bb7" containerName="istio-proxy" Jul 28 22:38:32.323984 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:32.323947 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e293242-3fc2-4c3b-9565-cf572c033bb7" containerName="istio-proxy" Jul 28 22:38:32.324145 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:32.324007 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e293242-3fc2-4c3b-9565-cf572c033bb7" containerName="istio-proxy" Jul 28 22:38:32.327378 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:32.327358 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b9e632fc-p24tk" Jul 28 22:38:32.329977 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:32.329947 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qf46x\"" Jul 28 22:38:32.335409 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:32.335383 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b9e632fc-p24tk"] Jul 28 22:38:32.408491 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:32.408459 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ce4cb68-f845-48ba-9570-8fe9235e7d0f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b9e632fc-p24tk\" (UID: \"2ce4cb68-f845-48ba-9570-8fe9235e7d0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b9e632fc-p24tk" Jul 28 22:38:32.509892 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:32.509854 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ce4cb68-f845-48ba-9570-8fe9235e7d0f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b9e632fc-p24tk\" (UID: \"2ce4cb68-f845-48ba-9570-8fe9235e7d0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b9e632fc-p24tk" Jul 28 22:38:32.510471 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:32.510451 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ce4cb68-f845-48ba-9570-8fe9235e7d0f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b9e632fc-p24tk\" (UID: \"2ce4cb68-f845-48ba-9570-8fe9235e7d0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b9e632fc-p24tk" Jul 28 22:38:32.642014 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:32.641930 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b9e632fc-p24tk" Jul 28 22:38:32.783540 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:32.783508 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b9e632fc-p24tk"] Jul 28 22:38:32.785879 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:38:32.785854 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ce4cb68_f845_48ba_9570_8fe9235e7d0f.slice/crio-a04346a50f81b83d97909a1db1f191263f4ddd39fe5c7b2af56b2a7b2826fd8d WatchSource:0}: Error finding container a04346a50f81b83d97909a1db1f191263f4ddd39fe5c7b2af56b2a7b2826fd8d: Status 404 returned error can't find the container with id a04346a50f81b83d97909a1db1f191263f4ddd39fe5c7b2af56b2a7b2826fd8d Jul 28 22:38:33.240376 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:33.240339 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b9e632fc-p24tk" event={"ID":"2ce4cb68-f845-48ba-9570-8fe9235e7d0f","Type":"ContainerStarted","Data":"000772f6a0d14c1eb05391df376d2f263786cf9d7b3bd4c9cbe8fbbff888d527"} Jul 28 22:38:33.240564 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:33.240398 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b9e632fc-p24tk" event={"ID":"2ce4cb68-f845-48ba-9570-8fe9235e7d0f","Type":"ContainerStarted","Data":"a04346a50f81b83d97909a1db1f191263f4ddd39fe5c7b2af56b2a7b2826fd8d"} Jul 28 22:38:33.262234 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:33.262180 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b9e632fc-p24tk" podStartSLOduration=1.262163929 podStartE2EDuration="1.262163929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:38:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:38:33.259783395 +0000 UTC m=+1733.347041464" watchObservedRunningTime="2026-07-28 22:38:33.262163929 +0000 UTC m=+1733.349421997" Jul 28 22:38:34.246499 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:34.246461 2553 generic.go:356] "Generic (PLEG): container finished" podID="2ce4cb68-f845-48ba-9570-8fe9235e7d0f" containerID="000772f6a0d14c1eb05391df376d2f263786cf9d7b3bd4c9cbe8fbbff888d527" exitCode=0 Jul 28 22:38:34.246945 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:34.246552 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b9e632fc-p24tk" event={"ID":"2ce4cb68-f845-48ba-9570-8fe9235e7d0f","Type":"ContainerDied","Data":"000772f6a0d14c1eb05391df376d2f263786cf9d7b3bd4c9cbe8fbbff888d527"} Jul 28 22:38:35.393655 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:35.393632 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b9e632fc-p24tk" Jul 28 22:38:35.437858 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:35.437830 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ce4cb68-f845-48ba-9570-8fe9235e7d0f-maas-api-service-ca\") pod \"2ce4cb68-f845-48ba-9570-8fe9235e7d0f\" (UID: \"2ce4cb68-f845-48ba-9570-8fe9235e7d0f\") " Jul 28 22:38:35.438271 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:35.438248 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ce4cb68-f845-48ba-9570-8fe9235e7d0f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2ce4cb68-f845-48ba-9570-8fe9235e7d0f" (UID: "2ce4cb68-f845-48ba-9570-8fe9235e7d0f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 22:38:35.539161 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:35.539074 2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ce4cb68-f845-48ba-9570-8fe9235e7d0f-maas-api-service-ca\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:38:36.259902 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:36.259869 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b9e632fc-p24tk" event={"ID":"2ce4cb68-f845-48ba-9570-8fe9235e7d0f","Type":"ContainerDied","Data":"a04346a50f81b83d97909a1db1f191263f4ddd39fe5c7b2af56b2a7b2826fd8d"} Jul 28 22:38:36.260081 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:36.259911 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b9e632fc-p24tk" Jul 28 22:38:36.260081 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:36.259917 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a04346a50f81b83d97909a1db1f191263f4ddd39fe5c7b2af56b2a7b2826fd8d" Jul 28 22:38:37.742108 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:37.742076 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b9e632fc-p24tk"] Jul 28 22:38:37.747153 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:37.747124 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b9e632fc-p24tk"] Jul 28 22:38:37.872041 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:37.871916 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml"] Jul 28 22:38:37.872300 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:37.872265 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml" podUID="9b4538f7-19c8-48cb-aebb-3f9666f9347f" containerName="istio-proxy" containerID="cri-o://2a7bbdca3305341b238e9fe53f0f15230ccfbe4d35f7a4486eaba2a8e7577470" gracePeriod=30 Jul 28 22:38:38.404988 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:38.404958 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ce4cb68-f845-48ba-9570-8fe9235e7d0f" path="/var/lib/kubelet/pods/2ce4cb68-f845-48ba-9570-8fe9235e7d0f/volumes" Jul 28 22:38:41.451101 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:41.451065 2553 patch_prober.go:28] interesting pod/e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 28 22:38:41.451472 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:41.451127 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml" podUID="9b4538f7-19c8-48cb-aebb-3f9666f9347f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 22:38:43.160592 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:43.160564 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml" Jul 28 22:38:43.209332 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:43.209296 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b4538f7-19c8-48cb-aebb-3f9666f9347f-credential-socket\") pod \"9b4538f7-19c8-48cb-aebb-3f9666f9347f\" (UID: \"9b4538f7-19c8-48cb-aebb-3f9666f9347f\") " Jul 28 22:38:43.209521 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:43.209342 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b4538f7-19c8-48cb-aebb-3f9666f9347f-istio-data\") pod \"9b4538f7-19c8-48cb-aebb-3f9666f9347f\" (UID: \"9b4538f7-19c8-48cb-aebb-3f9666f9347f\") " Jul 28 22:38:43.209521 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:43.209363 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b4538f7-19c8-48cb-aebb-3f9666f9347f-workload-socket\") pod \"9b4538f7-19c8-48cb-aebb-3f9666f9347f\" (UID: \"9b4538f7-19c8-48cb-aebb-3f9666f9347f\") " Jul 28 22:38:43.209521 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:43.209383 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b4538f7-19c8-48cb-aebb-3f9666f9347f-istiod-ca-cert\") pod \"9b4538f7-19c8-48cb-aebb-3f9666f9347f\" (UID: \"9b4538f7-19c8-48cb-aebb-3f9666f9347f\") " Jul 28 22:38:43.209521 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:43.209429 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mgtc6\" (UniqueName: \"kubernetes.io/projected/9b4538f7-19c8-48cb-aebb-3f9666f9347f-kube-api-access-mgtc6\") pod \"9b4538f7-19c8-48cb-aebb-3f9666f9347f\" (UID: \"9b4538f7-19c8-48cb-aebb-3f9666f9347f\") " Jul 28 22:38:43.209521 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:43.209456 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b4538f7-19c8-48cb-aebb-3f9666f9347f-istio-token\") pod \"9b4538f7-19c8-48cb-aebb-3f9666f9347f\" (UID: \"9b4538f7-19c8-48cb-aebb-3f9666f9347f\") " Jul 28 22:38:43.209809 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:43.209549 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b4538f7-19c8-48cb-aebb-3f9666f9347f-istio-envoy\") pod \"9b4538f7-19c8-48cb-aebb-3f9666f9347f\" (UID: \"9b4538f7-19c8-48cb-aebb-3f9666f9347f\") " Jul 28 22:38:43.209809 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:43.209565 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b4538f7-19c8-48cb-aebb-3f9666f9347f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9b4538f7-19c8-48cb-aebb-3f9666f9347f" (UID: "9b4538f7-19c8-48cb-aebb-3f9666f9347f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:38:43.209809 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:43.209585 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b4538f7-19c8-48cb-aebb-3f9666f9347f-workload-certs\") pod \"9b4538f7-19c8-48cb-aebb-3f9666f9347f\" (UID: \"9b4538f7-19c8-48cb-aebb-3f9666f9347f\") " Jul 28 22:38:43.209809 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:43.209635 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b4538f7-19c8-48cb-aebb-3f9666f9347f-istio-podinfo\") pod \"9b4538f7-19c8-48cb-aebb-3f9666f9347f\" (UID: \"9b4538f7-19c8-48cb-aebb-3f9666f9347f\") " Jul 28 22:38:43.209809 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:43.209763 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b4538f7-19c8-48cb-aebb-3f9666f9347f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9b4538f7-19c8-48cb-aebb-3f9666f9347f" (UID: "9b4538f7-19c8-48cb-aebb-3f9666f9347f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:38:43.210056 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:43.209845 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b4538f7-19c8-48cb-aebb-3f9666f9347f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9b4538f7-19c8-48cb-aebb-3f9666f9347f" (UID: "9b4538f7-19c8-48cb-aebb-3f9666f9347f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:38:43.210056 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:43.209891 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b4538f7-19c8-48cb-aebb-3f9666f9347f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9b4538f7-19c8-48cb-aebb-3f9666f9347f" (UID: "9b4538f7-19c8-48cb-aebb-3f9666f9347f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 22:38:43.210056 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:43.210013 2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b4538f7-19c8-48cb-aebb-3f9666f9347f-workload-certs\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:38:43.210056 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:43.210036 2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b4538f7-19c8-48cb-aebb-3f9666f9347f-credential-socket\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:38:43.210056 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:43.210051 2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b4538f7-19c8-48cb-aebb-3f9666f9347f-workload-socket\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:38:43.210244 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:43.210064 2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b4538f7-19c8-48cb-aebb-3f9666f9347f-istiod-ca-cert\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:38:43.210373 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:43.210224 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b4538f7-19c8-48cb-aebb-3f9666f9347f-istio-data" (OuterVolumeSpecName: "istio-data") pod "9b4538f7-19c8-48cb-aebb-3f9666f9347f" (UID: "9b4538f7-19c8-48cb-aebb-3f9666f9347f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:38:43.211888 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:43.211868 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b4538f7-19c8-48cb-aebb-3f9666f9347f-kube-api-access-mgtc6" (OuterVolumeSpecName: "kube-api-access-mgtc6") pod "9b4538f7-19c8-48cb-aebb-3f9666f9347f" (UID: "9b4538f7-19c8-48cb-aebb-3f9666f9347f"). InnerVolumeSpecName "kube-api-access-mgtc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:38:43.211888 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:43.211879 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b4538f7-19c8-48cb-aebb-3f9666f9347f-istio-token" (OuterVolumeSpecName: "istio-token") pod "9b4538f7-19c8-48cb-aebb-3f9666f9347f" (UID: "9b4538f7-19c8-48cb-aebb-3f9666f9347f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:38:43.212448 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:43.212430 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b4538f7-19c8-48cb-aebb-3f9666f9347f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9b4538f7-19c8-48cb-aebb-3f9666f9347f" (UID: "9b4538f7-19c8-48cb-aebb-3f9666f9347f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 22:38:43.212448 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:43.212434 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b4538f7-19c8-48cb-aebb-3f9666f9347f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9b4538f7-19c8-48cb-aebb-3f9666f9347f" (UID: "9b4538f7-19c8-48cb-aebb-3f9666f9347f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:38:43.302862 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:43.302829 2553 generic.go:356] "Generic (PLEG): container finished" podID="9b4538f7-19c8-48cb-aebb-3f9666f9347f" containerID="2a7bbdca3305341b238e9fe53f0f15230ccfbe4d35f7a4486eaba2a8e7577470" exitCode=0 Jul 28 22:38:43.303024 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:43.302877 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml" Jul 28 22:38:43.303024 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:43.302914 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml" event={"ID":"9b4538f7-19c8-48cb-aebb-3f9666f9347f","Type":"ContainerDied","Data":"2a7bbdca3305341b238e9fe53f0f15230ccfbe4d35f7a4486eaba2a8e7577470"} Jul 28 22:38:43.303024 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:43.302968 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml" event={"ID":"9b4538f7-19c8-48cb-aebb-3f9666f9347f","Type":"ContainerDied","Data":"8943e8b5c0648e7f2bd86b20d63e97e85732ffef5d87c5b90e7591db6e86499b"} Jul 28 22:38:43.303024 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:43.302989 2553 scope.go:117] "RemoveContainer" containerID="2a7bbdca3305341b238e9fe53f0f15230ccfbe4d35f7a4486eaba2a8e7577470" Jul 28 22:38:43.310566 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:43.310533 2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b4538f7-19c8-48cb-aebb-3f9666f9347f-istio-envoy\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:38:43.310566 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:43.310554 2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b4538f7-19c8-48cb-aebb-3f9666f9347f-istio-podinfo\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:38:43.310691 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:43.310568 2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b4538f7-19c8-48cb-aebb-3f9666f9347f-istio-data\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:38:43.310691 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:43.310583 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgtc6\" (UniqueName: \"kubernetes.io/projected/9b4538f7-19c8-48cb-aebb-3f9666f9347f-kube-api-access-mgtc6\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:38:43.310691 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:43.310598 2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b4538f7-19c8-48cb-aebb-3f9666f9347f-istio-token\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:38:43.312793 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:43.312775 2553 scope.go:117] "RemoveContainer" containerID="41438027eb2a127149e35d91260d73283464775073304ccf9a962ca6c2a77341" Jul 28 22:38:43.326098 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:43.326068 2553 scope.go:117] "RemoveContainer" containerID="2a7bbdca3305341b238e9fe53f0f15230ccfbe4d35f7a4486eaba2a8e7577470" Jul 28 22:38:43.326468 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:38:43.326442 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a7bbdca3305341b238e9fe53f0f15230ccfbe4d35f7a4486eaba2a8e7577470\": container with ID starting with 2a7bbdca3305341b238e9fe53f0f15230ccfbe4d35f7a4486eaba2a8e7577470 not found: ID does not exist" containerID="2a7bbdca3305341b238e9fe53f0f15230ccfbe4d35f7a4486eaba2a8e7577470" Jul 28 22:38:43.326556 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:43.326477 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a7bbdca3305341b238e9fe53f0f15230ccfbe4d35f7a4486eaba2a8e7577470"} err="failed to get container status \"2a7bbdca3305341b238e9fe53f0f15230ccfbe4d35f7a4486eaba2a8e7577470\": rpc error: code = NotFound desc = could not find container \"2a7bbdca3305341b238e9fe53f0f15230ccfbe4d35f7a4486eaba2a8e7577470\": container with ID starting with 2a7bbdca3305341b238e9fe53f0f15230ccfbe4d35f7a4486eaba2a8e7577470 not found: ID does not exist" Jul 28 22:38:43.326556 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:43.326495 2553 scope.go:117] "RemoveContainer" containerID="41438027eb2a127149e35d91260d73283464775073304ccf9a962ca6c2a77341" Jul 28 22:38:43.326913 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:38:43.326897 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41438027eb2a127149e35d91260d73283464775073304ccf9a962ca6c2a77341\": container with ID starting with 41438027eb2a127149e35d91260d73283464775073304ccf9a962ca6c2a77341 not found: ID does not exist" containerID="41438027eb2a127149e35d91260d73283464775073304ccf9a962ca6c2a77341" Jul 28 22:38:43.326984 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:43.326918 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41438027eb2a127149e35d91260d73283464775073304ccf9a962ca6c2a77341"} err="failed to get container status \"41438027eb2a127149e35d91260d73283464775073304ccf9a962ca6c2a77341\": rpc error: code = NotFound desc = could not find container \"41438027eb2a127149e35d91260d73283464775073304ccf9a962ca6c2a77341\": container with ID starting with 41438027eb2a127149e35d91260d73283464775073304ccf9a962ca6c2a77341 not found: ID does not exist" Jul 28 22:38:43.327928 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:43.327906 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml"] Jul 28 22:38:43.333648 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:43.333625 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-b9e632fc-openshift-default-647b9b464d-ncbml"] Jul 28 22:38:44.405003 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:38:44.404967 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b4538f7-19c8-48cb-aebb-3f9666f9347f" path="/var/lib/kubelet/pods/9b4538f7-19c8-48cb-aebb-3f9666f9347f/volumes" Jul 28 22:39:40.456279 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:39:40.456245 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mldhp_e1ced996-f136-4d3e-9c85-a5b7809b4517/console-operator/1.log" Jul 28 22:39:40.459178 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:39:40.459154 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mldhp_e1ced996-f136-4d3e-9c85-a5b7809b4517/console-operator/1.log" Jul 28 22:40:25.007575 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:25.007529 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px"] Jul 28 22:40:25.008071 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:25.007927 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b4538f7-19c8-48cb-aebb-3f9666f9347f" containerName="istio-proxy" Jul 28 22:40:25.008071 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:25.007938 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b4538f7-19c8-48cb-aebb-3f9666f9347f" containerName="istio-proxy" Jul 28 22:40:25.008071 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:25.007948 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b4538f7-19c8-48cb-aebb-3f9666f9347f" containerName="istio-proxy" Jul 28 22:40:25.008071 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:25.007953 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b4538f7-19c8-48cb-aebb-3f9666f9347f" containerName="istio-proxy" Jul 28 22:40:25.008071 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:25.007962 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ce4cb68-f845-48ba-9570-8fe9235e7d0f" containerName="revoke-keys" Jul 28 22:40:25.008071 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:25.007967 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ce4cb68-f845-48ba-9570-8fe9235e7d0f" containerName="revoke-keys" Jul 28 22:40:25.008071 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:25.008028 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e293242-3fc2-4c3b-9565-cf572c033bb7" containerName="istio-proxy" Jul 28 22:40:25.008071 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:25.008038 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b4538f7-19c8-48cb-aebb-3f9666f9347f" containerName="istio-proxy" Jul 28 22:40:25.008071 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:25.008045 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ce4cb68-f845-48ba-9570-8fe9235e7d0f" containerName="revoke-keys" Jul 28 22:40:25.008071 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:25.008051 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b4538f7-19c8-48cb-aebb-3f9666f9347f" containerName="istio-proxy" Jul 28 22:40:25.010420 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:25.010401 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px" Jul 28 22:40:25.014587 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:25.014567 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-b45d9e9d-openshift-default-dockercfg-w58cr\"" Jul 28 22:40:25.046520 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:25.046494 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px"] Jul 28 22:40:25.091959 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:25.091920 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f691f0-3715-4758-9928-2cf425ae448f-workload-socket\") pod \"e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px\" (UID: \"26f691f0-3715-4758-9928-2cf425ae448f\") " pod="openshift-ingress/e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px" Jul 28 22:40:25.091959 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:25.091965 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26f691f0-3715-4758-9928-2cf425ae448f-workload-certs\") pod \"e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px\" (UID: \"26f691f0-3715-4758-9928-2cf425ae448f\") " pod="openshift-ingress/e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px" Jul 28 22:40:25.092220 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:25.091990 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26f691f0-3715-4758-9928-2cf425ae448f-istiod-ca-cert\") pod \"e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px\" (UID: \"26f691f0-3715-4758-9928-2cf425ae448f\") " pod="openshift-ingress/e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px" Jul 28 22:40:25.092220 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:25.092017 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26f691f0-3715-4758-9928-2cf425ae448f-istio-envoy\") pod \"e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px\" (UID: \"26f691f0-3715-4758-9928-2cf425ae448f\") " pod="openshift-ingress/e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px" Jul 28 22:40:25.092220 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:25.092052 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26f691f0-3715-4758-9928-2cf425ae448f-istio-data\") pod \"e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px\" (UID: \"26f691f0-3715-4758-9928-2cf425ae448f\") " pod="openshift-ingress/e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px" Jul 28 22:40:25.092220 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:25.092119 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxl9x\" (UniqueName: \"kubernetes.io/projected/26f691f0-3715-4758-9928-2cf425ae448f-kube-api-access-zxl9x\") pod \"e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px\" (UID: \"26f691f0-3715-4758-9928-2cf425ae448f\") " pod="openshift-ingress/e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px" Jul 28 22:40:25.092220 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:25.092149 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26f691f0-3715-4758-9928-2cf425ae448f-istio-podinfo\") pod \"e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px\" (UID: \"26f691f0-3715-4758-9928-2cf425ae448f\") " pod="openshift-ingress/e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px" Jul 28 22:40:25.092220 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:25.092171 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26f691f0-3715-4758-9928-2cf425ae448f-istio-token\") pod \"e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px\" (UID: \"26f691f0-3715-4758-9928-2cf425ae448f\") " pod="openshift-ingress/e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px" Jul 28 22:40:25.092220 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:25.092202 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f691f0-3715-4758-9928-2cf425ae448f-credential-socket\") pod \"e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px\" (UID: \"26f691f0-3715-4758-9928-2cf425ae448f\") " pod="openshift-ingress/e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px" Jul 28 22:40:25.192994 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:25.192964 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f691f0-3715-4758-9928-2cf425ae448f-workload-socket\") pod \"e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px\" (UID: \"26f691f0-3715-4758-9928-2cf425ae448f\") " pod="openshift-ingress/e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px" Jul 28 22:40:25.192994 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:25.192997 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26f691f0-3715-4758-9928-2cf425ae448f-workload-certs\") pod \"e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px\" (UID: \"26f691f0-3715-4758-9928-2cf425ae448f\") " pod="openshift-ingress/e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px" Jul 28 22:40:25.193252 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:25.193016 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26f691f0-3715-4758-9928-2cf425ae448f-istiod-ca-cert\") pod \"e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px\" (UID: \"26f691f0-3715-4758-9928-2cf425ae448f\") " pod="openshift-ingress/e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px" Jul 28 22:40:25.193252 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:25.193033 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26f691f0-3715-4758-9928-2cf425ae448f-istio-envoy\") pod \"e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px\" (UID: \"26f691f0-3715-4758-9928-2cf425ae448f\") " pod="openshift-ingress/e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px" Jul 28 22:40:25.193252 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:25.193056 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26f691f0-3715-4758-9928-2cf425ae448f-istio-data\") pod \"e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px\" (UID: \"26f691f0-3715-4758-9928-2cf425ae448f\") " pod="openshift-ingress/e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px" Jul 28 22:40:25.193252 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:25.193094 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxl9x\" (UniqueName: \"kubernetes.io/projected/26f691f0-3715-4758-9928-2cf425ae448f-kube-api-access-zxl9x\") pod \"e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px\" (UID: \"26f691f0-3715-4758-9928-2cf425ae448f\") " pod="openshift-ingress/e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px" Jul 28 22:40:25.193252 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:25.193113 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26f691f0-3715-4758-9928-2cf425ae448f-istio-podinfo\") pod \"e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px\" (UID: \"26f691f0-3715-4758-9928-2cf425ae448f\") " pod="openshift-ingress/e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px" Jul 28 22:40:25.193252 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:25.193130 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26f691f0-3715-4758-9928-2cf425ae448f-istio-token\") pod \"e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px\" (UID: \"26f691f0-3715-4758-9928-2cf425ae448f\") " pod="openshift-ingress/e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px" Jul 28 22:40:25.193252 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:25.193148 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f691f0-3715-4758-9928-2cf425ae448f-credential-socket\") pod \"e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px\" (UID: \"26f691f0-3715-4758-9928-2cf425ae448f\") " pod="openshift-ingress/e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px" Jul 28 22:40:25.193603 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:25.193430 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26f691f0-3715-4758-9928-2cf425ae448f-workload-certs\") pod \"e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px\" (UID: \"26f691f0-3715-4758-9928-2cf425ae448f\") " pod="openshift-ingress/e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px" Jul 28 22:40:25.193603 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:25.193454 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f691f0-3715-4758-9928-2cf425ae448f-workload-socket\") pod \"e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px\" (UID: \"26f691f0-3715-4758-9928-2cf425ae448f\") " pod="openshift-ingress/e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px" Jul 28 22:40:25.193603 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:25.193571 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f691f0-3715-4758-9928-2cf425ae448f-credential-socket\") pod \"e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px\" (UID: \"26f691f0-3715-4758-9928-2cf425ae448f\") " pod="openshift-ingress/e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px" Jul 28 22:40:25.193764 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:25.193690 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26f691f0-3715-4758-9928-2cf425ae448f-istio-data\") pod \"e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px\" (UID: \"26f691f0-3715-4758-9928-2cf425ae448f\") " pod="openshift-ingress/e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px" Jul 28 22:40:25.193946 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:25.193926 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26f691f0-3715-4758-9928-2cf425ae448f-istiod-ca-cert\") pod \"e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px\" (UID: \"26f691f0-3715-4758-9928-2cf425ae448f\") " pod="openshift-ingress/e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px" Jul 28 22:40:25.195732 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:25.195696 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26f691f0-3715-4758-9928-2cf425ae448f-istio-envoy\") pod \"e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px\" (UID: \"26f691f0-3715-4758-9928-2cf425ae448f\") " pod="openshift-ingress/e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px" Jul 28 22:40:25.195892 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:25.195863 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26f691f0-3715-4758-9928-2cf425ae448f-istio-podinfo\") pod \"e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px\" (UID: \"26f691f0-3715-4758-9928-2cf425ae448f\") " pod="openshift-ingress/e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px" Jul 28 22:40:25.206467 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:25.206440 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxl9x\" (UniqueName: \"kubernetes.io/projected/26f691f0-3715-4758-9928-2cf425ae448f-kube-api-access-zxl9x\") pod \"e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px\" (UID: \"26f691f0-3715-4758-9928-2cf425ae448f\") " pod="openshift-ingress/e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px" Jul 28 22:40:25.216383 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:25.216360 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26f691f0-3715-4758-9928-2cf425ae448f-istio-token\") pod \"e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px\" (UID: \"26f691f0-3715-4758-9928-2cf425ae448f\") " pod="openshift-ingress/e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px" Jul 28 22:40:25.321434 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:25.321336 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px" Jul 28 22:40:25.462584 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:25.462558 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px"] Jul 28 22:40:25.464793 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:40:25.464768 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26f691f0_3715_4758_9928_2cf425ae448f.slice/crio-1a5b1c15e3e30b15e1571697dba1ec1a744fdfe24359bff007080a5228d7f2a4 WatchSource:0}: Error finding container 1a5b1c15e3e30b15e1571697dba1ec1a744fdfe24359bff007080a5228d7f2a4: Status 404 returned error can't find the container with id 1a5b1c15e3e30b15e1571697dba1ec1a744fdfe24359bff007080a5228d7f2a4 Jul 28 22:40:25.466735 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:25.466684 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:40:25.466798 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:25.466774 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:40:25.466836 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:25.466803 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:40:25.843967 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:25.843928 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px" event={"ID":"26f691f0-3715-4758-9928-2cf425ae448f","Type":"ContainerStarted","Data":"1b5e0653f0884772e77d73a25f232d29add28c62b5c779c2748d76dd057124b6"} Jul 28 22:40:25.844149 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:25.844019 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px" event={"ID":"26f691f0-3715-4758-9928-2cf425ae448f","Type":"ContainerStarted","Data":"1a5b1c15e3e30b15e1571697dba1ec1a744fdfe24359bff007080a5228d7f2a4"} Jul 28 22:40:25.866399 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:25.866338 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px" podStartSLOduration=1.86631874 podStartE2EDuration="1.86631874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:40:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:40:25.862839023 +0000 UTC m=+1845.950097094" watchObservedRunningTime="2026-07-28 22:40:25.86631874 +0000 UTC m=+1845.953576813" Jul 28 22:40:26.321652 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:26.321612 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px" Jul 28 22:40:26.326429 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:26.326401 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px" Jul 28 22:40:26.849212 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:26.849183 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px" Jul 28 22:40:26.850334 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:26.850312 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px" Jul 28 22:40:27.873809 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:27.873777 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b45d9e9d-75d47dc969-cmbtf"] Jul 28 22:40:27.877317 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:27.877301 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b45d9e9d-75d47dc969-cmbtf" Jul 28 22:40:27.882981 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:27.882956 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-b45d9e9d-serving-cert\"" Jul 28 22:40:27.895682 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:27.895657 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b45d9e9d-75d47dc969-cmbtf"] Jul 28 22:40:28.019404 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:28.019368 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mftm\" (UniqueName: \"kubernetes.io/projected/75f8991f-441b-462a-a1a5-377b9134774e-kube-api-access-7mftm\") pod \"maas-api-e2e-mt-b45d9e9d-75d47dc969-cmbtf\" (UID: \"75f8991f-441b-462a-a1a5-377b9134774e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b45d9e9d-75d47dc969-cmbtf" Jul 28 22:40:28.019582 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:28.019428 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75f8991f-441b-462a-a1a5-377b9134774e-openshift-service-ca\") pod \"maas-api-e2e-mt-b45d9e9d-75d47dc969-cmbtf\" (UID: \"75f8991f-441b-462a-a1a5-377b9134774e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b45d9e9d-75d47dc969-cmbtf" Jul 28 22:40:28.019582 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:28.019470 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75f8991f-441b-462a-a1a5-377b9134774e-maas-api-tls\") pod \"maas-api-e2e-mt-b45d9e9d-75d47dc969-cmbtf\" (UID: \"75f8991f-441b-462a-a1a5-377b9134774e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b45d9e9d-75d47dc969-cmbtf" Jul 28 22:40:28.120074 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:28.120042 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mftm\" (UniqueName: \"kubernetes.io/projected/75f8991f-441b-462a-a1a5-377b9134774e-kube-api-access-7mftm\") pod \"maas-api-e2e-mt-b45d9e9d-75d47dc969-cmbtf\" (UID: \"75f8991f-441b-462a-a1a5-377b9134774e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b45d9e9d-75d47dc969-cmbtf" Jul 28 22:40:28.120256 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:28.120114 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75f8991f-441b-462a-a1a5-377b9134774e-openshift-service-ca\") pod \"maas-api-e2e-mt-b45d9e9d-75d47dc969-cmbtf\" (UID: \"75f8991f-441b-462a-a1a5-377b9134774e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b45d9e9d-75d47dc969-cmbtf" Jul 28 22:40:28.120256 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:28.120152 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75f8991f-441b-462a-a1a5-377b9134774e-maas-api-tls\") pod \"maas-api-e2e-mt-b45d9e9d-75d47dc969-cmbtf\" (UID: \"75f8991f-441b-462a-a1a5-377b9134774e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b45d9e9d-75d47dc969-cmbtf" Jul 28 22:40:28.120836 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:28.120817 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75f8991f-441b-462a-a1a5-377b9134774e-openshift-service-ca\") pod \"maas-api-e2e-mt-b45d9e9d-75d47dc969-cmbtf\" (UID: \"75f8991f-441b-462a-a1a5-377b9134774e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b45d9e9d-75d47dc969-cmbtf" Jul 28 22:40:28.122799 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:28.122778 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75f8991f-441b-462a-a1a5-377b9134774e-maas-api-tls\") pod \"maas-api-e2e-mt-b45d9e9d-75d47dc969-cmbtf\" (UID: \"75f8991f-441b-462a-a1a5-377b9134774e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b45d9e9d-75d47dc969-cmbtf" Jul 28 22:40:28.128877 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:28.128807 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mftm\" (UniqueName: \"kubernetes.io/projected/75f8991f-441b-462a-a1a5-377b9134774e-kube-api-access-7mftm\") pod \"maas-api-e2e-mt-b45d9e9d-75d47dc969-cmbtf\" (UID: \"75f8991f-441b-462a-a1a5-377b9134774e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b45d9e9d-75d47dc969-cmbtf" Jul 28 22:40:28.189876 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:28.189843 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b45d9e9d-75d47dc969-cmbtf" Jul 28 22:40:28.331245 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:28.331221 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b45d9e9d-75d47dc969-cmbtf"] Jul 28 22:40:28.333185 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:40:28.333162 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75f8991f_441b_462a_a1a5_377b9134774e.slice/crio-e28bde484ac7d24d2a67c0ae53a9b5434f9bdb263edd02d9bd3bdd690cb67ff1 WatchSource:0}: Error finding container e28bde484ac7d24d2a67c0ae53a9b5434f9bdb263edd02d9bd3bdd690cb67ff1: Status 404 returned error can't find the container with id e28bde484ac7d24d2a67c0ae53a9b5434f9bdb263edd02d9bd3bdd690cb67ff1 Jul 28 22:40:28.860024 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:28.859989 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b45d9e9d-75d47dc969-cmbtf" event={"ID":"75f8991f-441b-462a-a1a5-377b9134774e","Type":"ContainerStarted","Data":"e28bde484ac7d24d2a67c0ae53a9b5434f9bdb263edd02d9bd3bdd690cb67ff1"} Jul 28 22:40:29.867756 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:29.867684 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b45d9e9d-75d47dc969-cmbtf" event={"ID":"75f8991f-441b-462a-a1a5-377b9134774e","Type":"ContainerStarted","Data":"1a13b29a594b9030af52af5fc7b6443b0ec2970ade4bd9f15d26b0dae3819f41"} Jul 28 22:40:29.868394 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:29.867776 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b45d9e9d-75d47dc969-cmbtf" Jul 28 22:40:29.887955 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:29.887900 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b45d9e9d-75d47dc969-cmbtf" podStartSLOduration=2.34824079 podStartE2EDuration="2.887880006s" totalImagesPullingTime="539.639216ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:40:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:40:29.887527712 +0000 UTC m=+1849.974785780" watchObservedRunningTime="2026-07-28 22:40:29.887880006 +0000 UTC m=+1849.975138077" Jul 28 22:40:35.879118 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:35.879091 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b45d9e9d-75d47dc969-cmbtf" Jul 28 22:40:42.165563 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:42.164353 2553 patch_prober.go:28] interesting pod/e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": read tcp 10.132.0.2:44834->10.132.0.53:15021: read: connection reset by peer" start-of-body= Jul 28 22:40:42.165563 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:42.164422 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px" podUID="26f691f0-3715-4758-9928-2cf425ae448f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": read tcp 10.132.0.2:44834->10.132.0.53:15021: read: connection reset by peer" Jul 28 22:40:42.942648 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:42.942616 2553 generic.go:356] "Generic (PLEG): container finished" podID="26f691f0-3715-4758-9928-2cf425ae448f" containerID="1b5e0653f0884772e77d73a25f232d29add28c62b5c779c2748d76dd057124b6" exitCode=0 Jul 28 22:40:42.942852 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:42.942693 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px" event={"ID":"26f691f0-3715-4758-9928-2cf425ae448f","Type":"ContainerDied","Data":"1b5e0653f0884772e77d73a25f232d29add28c62b5c779c2748d76dd057124b6"} Jul 28 22:40:42.943117 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:42.943104 2553 scope.go:117] "RemoveContainer" containerID="1b5e0653f0884772e77d73a25f232d29add28c62b5c779c2748d76dd057124b6" Jul 28 22:40:42.944120 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:42.944095 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:40:42.944168 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:42.944160 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:40:42.944206 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:42.944192 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:40:43.950061 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:43.950030 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px" event={"ID":"26f691f0-3715-4758-9928-2cf425ae448f","Type":"ContainerStarted","Data":"c858aad5e36542ffc2b7d90191ab32d68f754fe2248002012d7a275e9462b930"} Jul 28 22:40:44.322088 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:44.322046 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px" Jul 28 22:40:44.322502 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:44.322475 2553 patch_prober.go:28] interesting pod/e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 28 22:40:44.322697 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:44.322669 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px" podUID="26f691f0-3715-4758-9928-2cf425ae448f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 28 22:40:45.322887 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:45.322849 2553 patch_prober.go:28] interesting pod/e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 28 22:40:45.323481 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:45.323446 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px" podUID="26f691f0-3715-4758-9928-2cf425ae448f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 28 22:40:46.322655 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:46.322605 2553 patch_prober.go:28] interesting pod/e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 28 22:40:46.322893 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:46.322680 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px" podUID="26f691f0-3715-4758-9928-2cf425ae448f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 28 22:40:47.321834 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:47.321789 2553 patch_prober.go:28] interesting pod/e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 28 22:40:47.322350 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:47.321864 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px" podUID="26f691f0-3715-4758-9928-2cf425ae448f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 28 22:40:48.324893 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:48.324860 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px" Jul 28 22:40:48.325365 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:48.325231 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px" Jul 28 22:40:48.325772 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:40:48.325753 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px" Jul 28 22:41:10.463543 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:10.463513 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b45d9e9d-75d47dc969-cmbtf"] Jul 28 22:41:10.464164 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:10.463745 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b45d9e9d-75d47dc969-cmbtf" podUID="75f8991f-441b-462a-a1a5-377b9134774e" containerName="maas-api" containerID="cri-o://1a13b29a594b9030af52af5fc7b6443b0ec2970ade4bd9f15d26b0dae3819f41" gracePeriod=30 Jul 28 22:41:10.722860 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:10.722797 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b45d9e9d-75d47dc969-cmbtf" Jul 28 22:41:10.788416 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:10.788389 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75f8991f-441b-462a-a1a5-377b9134774e-maas-api-tls\") pod \"75f8991f-441b-462a-a1a5-377b9134774e\" (UID: \"75f8991f-441b-462a-a1a5-377b9134774e\") " Jul 28 22:41:10.788592 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:10.788466 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7mftm\" (UniqueName: \"kubernetes.io/projected/75f8991f-441b-462a-a1a5-377b9134774e-kube-api-access-7mftm\") pod \"75f8991f-441b-462a-a1a5-377b9134774e\" (UID: \"75f8991f-441b-462a-a1a5-377b9134774e\") " Jul 28 22:41:10.788665 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:10.788638 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75f8991f-441b-462a-a1a5-377b9134774e-openshift-service-ca\") pod \"75f8991f-441b-462a-a1a5-377b9134774e\" (UID: \"75f8991f-441b-462a-a1a5-377b9134774e\") " Jul 28 22:41:10.788953 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:10.788932 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75f8991f-441b-462a-a1a5-377b9134774e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "75f8991f-441b-462a-a1a5-377b9134774e" (UID: "75f8991f-441b-462a-a1a5-377b9134774e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 22:41:10.790562 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:10.790546 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75f8991f-441b-462a-a1a5-377b9134774e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "75f8991f-441b-462a-a1a5-377b9134774e" (UID: "75f8991f-441b-462a-a1a5-377b9134774e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 22:41:10.790662 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:10.790649 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75f8991f-441b-462a-a1a5-377b9134774e-kube-api-access-7mftm" (OuterVolumeSpecName: "kube-api-access-7mftm") pod "75f8991f-441b-462a-a1a5-377b9134774e" (UID: "75f8991f-441b-462a-a1a5-377b9134774e"). InnerVolumeSpecName "kube-api-access-7mftm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:41:10.889303 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:10.889273 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mftm\" (UniqueName: \"kubernetes.io/projected/75f8991f-441b-462a-a1a5-377b9134774e-kube-api-access-7mftm\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:41:10.889303 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:10.889297 2553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75f8991f-441b-462a-a1a5-377b9134774e-openshift-service-ca\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:41:10.889303 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:10.889307 2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75f8991f-441b-462a-a1a5-377b9134774e-maas-api-tls\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:41:11.096494 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:11.096407 2553 generic.go:356] "Generic (PLEG): container finished" podID="75f8991f-441b-462a-a1a5-377b9134774e" containerID="1a13b29a594b9030af52af5fc7b6443b0ec2970ade4bd9f15d26b0dae3819f41" exitCode=0 Jul 28 22:41:11.096494 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:11.096449 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b45d9e9d-75d47dc969-cmbtf" Jul 28 22:41:11.096726 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:11.096498 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b45d9e9d-75d47dc969-cmbtf" event={"ID":"75f8991f-441b-462a-a1a5-377b9134774e","Type":"ContainerDied","Data":"1a13b29a594b9030af52af5fc7b6443b0ec2970ade4bd9f15d26b0dae3819f41"} Jul 28 22:41:11.096726 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:11.096551 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b45d9e9d-75d47dc969-cmbtf" event={"ID":"75f8991f-441b-462a-a1a5-377b9134774e","Type":"ContainerDied","Data":"e28bde484ac7d24d2a67c0ae53a9b5434f9bdb263edd02d9bd3bdd690cb67ff1"} Jul 28 22:41:11.096726 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:11.096578 2553 scope.go:117] "RemoveContainer" containerID="1a13b29a594b9030af52af5fc7b6443b0ec2970ade4bd9f15d26b0dae3819f41" Jul 28 22:41:11.105874 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:11.105858 2553 scope.go:117] "RemoveContainer" containerID="1a13b29a594b9030af52af5fc7b6443b0ec2970ade4bd9f15d26b0dae3819f41" Jul 28 22:41:11.106137 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:41:11.106117 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a13b29a594b9030af52af5fc7b6443b0ec2970ade4bd9f15d26b0dae3819f41\": container with ID starting with 1a13b29a594b9030af52af5fc7b6443b0ec2970ade4bd9f15d26b0dae3819f41 not found: ID does not exist" containerID="1a13b29a594b9030af52af5fc7b6443b0ec2970ade4bd9f15d26b0dae3819f41" Jul 28 22:41:11.106188 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:11.106146 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a13b29a594b9030af52af5fc7b6443b0ec2970ade4bd9f15d26b0dae3819f41"} err="failed to get container status \"1a13b29a594b9030af52af5fc7b6443b0ec2970ade4bd9f15d26b0dae3819f41\": rpc error: code = NotFound desc = could not find container \"1a13b29a594b9030af52af5fc7b6443b0ec2970ade4bd9f15d26b0dae3819f41\": container with ID starting with 1a13b29a594b9030af52af5fc7b6443b0ec2970ade4bd9f15d26b0dae3819f41 not found: ID does not exist" Jul 28 22:41:11.117159 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:11.117132 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b45d9e9d-75d47dc969-cmbtf"] Jul 28 22:41:11.120734 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:11.120712 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b45d9e9d-75d47dc969-cmbtf"] Jul 28 22:41:12.405418 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:12.405384 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75f8991f-441b-462a-a1a5-377b9134774e" path="/var/lib/kubelet/pods/75f8991f-441b-462a-a1a5-377b9134774e/volumes" Jul 28 22:41:17.078609 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:17.078578 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px"] Jul 28 22:41:17.079189 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:17.078847 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px" podUID="26f691f0-3715-4758-9928-2cf425ae448f" containerName="istio-proxy" containerID="cri-o://c858aad5e36542ffc2b7d90191ab32d68f754fe2248002012d7a275e9462b930" gracePeriod=30 Jul 28 22:41:18.328263 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:18.328228 2553 patch_prober.go:28] interesting pod/e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Jul 28 22:41:18.328263 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:18.328279 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px" podUID="26f691f0-3715-4758-9928-2cf425ae448f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 22:41:22.343212 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:22.343147 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px" Jul 28 22:41:22.391110 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:22.391076 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zxl9x\" (UniqueName: \"kubernetes.io/projected/26f691f0-3715-4758-9928-2cf425ae448f-kube-api-access-zxl9x\") pod \"26f691f0-3715-4758-9928-2cf425ae448f\" (UID: \"26f691f0-3715-4758-9928-2cf425ae448f\") " Jul 28 22:41:22.391312 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:22.391140 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26f691f0-3715-4758-9928-2cf425ae448f-istio-data\") pod \"26f691f0-3715-4758-9928-2cf425ae448f\" (UID: \"26f691f0-3715-4758-9928-2cf425ae448f\") " Jul 28 22:41:22.391312 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:22.391165 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f691f0-3715-4758-9928-2cf425ae448f-credential-socket\") pod \"26f691f0-3715-4758-9928-2cf425ae448f\" (UID: \"26f691f0-3715-4758-9928-2cf425ae448f\") " Jul 28 22:41:22.391312 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:22.391179 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f691f0-3715-4758-9928-2cf425ae448f-workload-socket\") pod \"26f691f0-3715-4758-9928-2cf425ae448f\" (UID: \"26f691f0-3715-4758-9928-2cf425ae448f\") " Jul 28 22:41:22.391312 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:22.391205 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26f691f0-3715-4758-9928-2cf425ae448f-istio-podinfo\") pod \"26f691f0-3715-4758-9928-2cf425ae448f\" (UID: \"26f691f0-3715-4758-9928-2cf425ae448f\") " Jul 28 22:41:22.391312 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:22.391232 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26f691f0-3715-4758-9928-2cf425ae448f-istio-envoy\") pod \"26f691f0-3715-4758-9928-2cf425ae448f\" (UID: \"26f691f0-3715-4758-9928-2cf425ae448f\") " Jul 28 22:41:22.391312 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:22.391266 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26f691f0-3715-4758-9928-2cf425ae448f-istiod-ca-cert\") pod \"26f691f0-3715-4758-9928-2cf425ae448f\" (UID: \"26f691f0-3715-4758-9928-2cf425ae448f\") " Jul 28 22:41:22.391312 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:22.391310 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26f691f0-3715-4758-9928-2cf425ae448f-workload-certs\") pod \"26f691f0-3715-4758-9928-2cf425ae448f\" (UID: \"26f691f0-3715-4758-9928-2cf425ae448f\") " Jul 28 22:41:22.391677 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:22.391334 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26f691f0-3715-4758-9928-2cf425ae448f-istio-token\") pod \"26f691f0-3715-4758-9928-2cf425ae448f\" (UID: \"26f691f0-3715-4758-9928-2cf425ae448f\") " Jul 28 22:41:22.391677 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:22.391501 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26f691f0-3715-4758-9928-2cf425ae448f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "26f691f0-3715-4758-9928-2cf425ae448f" (UID: "26f691f0-3715-4758-9928-2cf425ae448f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:41:22.391677 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:22.391509 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26f691f0-3715-4758-9928-2cf425ae448f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "26f691f0-3715-4758-9928-2cf425ae448f" (UID: "26f691f0-3715-4758-9928-2cf425ae448f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:41:22.391677 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:22.391653 2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f691f0-3715-4758-9928-2cf425ae448f-credential-socket\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:41:22.391677 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:22.391674 2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f691f0-3715-4758-9928-2cf425ae448f-workload-socket\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:41:22.391971 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:22.391687 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26f691f0-3715-4758-9928-2cf425ae448f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "26f691f0-3715-4758-9928-2cf425ae448f" (UID: "26f691f0-3715-4758-9928-2cf425ae448f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:41:22.392078 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:22.392050 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26f691f0-3715-4758-9928-2cf425ae448f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "26f691f0-3715-4758-9928-2cf425ae448f" (UID: "26f691f0-3715-4758-9928-2cf425ae448f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 22:41:22.392396 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:22.392277 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26f691f0-3715-4758-9928-2cf425ae448f-istio-data" (OuterVolumeSpecName: "istio-data") pod "26f691f0-3715-4758-9928-2cf425ae448f" (UID: "26f691f0-3715-4758-9928-2cf425ae448f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:41:22.393883 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:22.393853 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26f691f0-3715-4758-9928-2cf425ae448f-istio-token" (OuterVolumeSpecName: "istio-token") pod "26f691f0-3715-4758-9928-2cf425ae448f" (UID: "26f691f0-3715-4758-9928-2cf425ae448f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:41:22.393883 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:22.393874 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26f691f0-3715-4758-9928-2cf425ae448f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "26f691f0-3715-4758-9928-2cf425ae448f" (UID: "26f691f0-3715-4758-9928-2cf425ae448f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:41:22.394039 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:22.394010 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26f691f0-3715-4758-9928-2cf425ae448f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "26f691f0-3715-4758-9928-2cf425ae448f" (UID: "26f691f0-3715-4758-9928-2cf425ae448f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 22:41:22.394253 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:22.394236 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26f691f0-3715-4758-9928-2cf425ae448f-kube-api-access-zxl9x" (OuterVolumeSpecName: "kube-api-access-zxl9x") pod "26f691f0-3715-4758-9928-2cf425ae448f" (UID: "26f691f0-3715-4758-9928-2cf425ae448f"). InnerVolumeSpecName "kube-api-access-zxl9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:41:22.492259 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:22.492218 2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26f691f0-3715-4758-9928-2cf425ae448f-istio-data\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:41:22.492259 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:22.492252 2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26f691f0-3715-4758-9928-2cf425ae448f-istio-podinfo\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:41:22.492259 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:22.492261 2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26f691f0-3715-4758-9928-2cf425ae448f-istio-envoy\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:41:22.492259 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:22.492271 2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26f691f0-3715-4758-9928-2cf425ae448f-istiod-ca-cert\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:41:22.492538 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:22.492280 2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26f691f0-3715-4758-9928-2cf425ae448f-workload-certs\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:41:22.492538 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:22.492287 2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26f691f0-3715-4758-9928-2cf425ae448f-istio-token\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:41:22.492538 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:22.492295 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxl9x\" (UniqueName: \"kubernetes.io/projected/26f691f0-3715-4758-9928-2cf425ae448f-kube-api-access-zxl9x\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:41:23.167091 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:23.167058 2553 generic.go:356] "Generic (PLEG): container finished" podID="26f691f0-3715-4758-9928-2cf425ae448f" containerID="c858aad5e36542ffc2b7d90191ab32d68f754fe2248002012d7a275e9462b930" exitCode=0 Jul 28 22:41:23.167270 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:23.167108 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px" Jul 28 22:41:23.167270 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:23.167138 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px" event={"ID":"26f691f0-3715-4758-9928-2cf425ae448f","Type":"ContainerDied","Data":"c858aad5e36542ffc2b7d90191ab32d68f754fe2248002012d7a275e9462b930"} Jul 28 22:41:23.167270 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:23.167194 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px" event={"ID":"26f691f0-3715-4758-9928-2cf425ae448f","Type":"ContainerDied","Data":"1a5b1c15e3e30b15e1571697dba1ec1a744fdfe24359bff007080a5228d7f2a4"} Jul 28 22:41:23.167270 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:23.167217 2553 scope.go:117] "RemoveContainer" containerID="c858aad5e36542ffc2b7d90191ab32d68f754fe2248002012d7a275e9462b930" Jul 28 22:41:23.178466 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:23.178415 2553 scope.go:117] "RemoveContainer" containerID="1b5e0653f0884772e77d73a25f232d29add28c62b5c779c2748d76dd057124b6" Jul 28 22:41:23.183945 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:23.183921 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px"] Jul 28 22:41:23.187441 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:23.187422 2553 scope.go:117] "RemoveContainer" containerID="c858aad5e36542ffc2b7d90191ab32d68f754fe2248002012d7a275e9462b930" Jul 28 22:41:23.187686 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:41:23.187666 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c858aad5e36542ffc2b7d90191ab32d68f754fe2248002012d7a275e9462b930\": container with ID starting with c858aad5e36542ffc2b7d90191ab32d68f754fe2248002012d7a275e9462b930 not found: ID does not exist" containerID="c858aad5e36542ffc2b7d90191ab32d68f754fe2248002012d7a275e9462b930" Jul 28 22:41:23.187867 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:23.187697 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c858aad5e36542ffc2b7d90191ab32d68f754fe2248002012d7a275e9462b930"} err="failed to get container status \"c858aad5e36542ffc2b7d90191ab32d68f754fe2248002012d7a275e9462b930\": rpc error: code = NotFound desc = could not find container \"c858aad5e36542ffc2b7d90191ab32d68f754fe2248002012d7a275e9462b930\": container with ID starting with c858aad5e36542ffc2b7d90191ab32d68f754fe2248002012d7a275e9462b930 not found: ID does not exist" Jul 28 22:41:23.187867 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:23.187742 2553 scope.go:117] "RemoveContainer" containerID="1b5e0653f0884772e77d73a25f232d29add28c62b5c779c2748d76dd057124b6" Jul 28 22:41:23.188008 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:41:23.187985 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b5e0653f0884772e77d73a25f232d29add28c62b5c779c2748d76dd057124b6\": container with ID starting with 1b5e0653f0884772e77d73a25f232d29add28c62b5c779c2748d76dd057124b6 not found: ID does not exist" containerID="1b5e0653f0884772e77d73a25f232d29add28c62b5c779c2748d76dd057124b6" Jul 28 22:41:23.188049 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:23.188021 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b5e0653f0884772e77d73a25f232d29add28c62b5c779c2748d76dd057124b6"} err="failed to get container status \"1b5e0653f0884772e77d73a25f232d29add28c62b5c779c2748d76dd057124b6\": rpc error: code = NotFound desc = could not find container \"1b5e0653f0884772e77d73a25f232d29add28c62b5c779c2748d76dd057124b6\": container with ID starting with 1b5e0653f0884772e77d73a25f232d29add28c62b5c779c2748d76dd057124b6 not found: ID does not exist" Jul 28 22:41:23.188049 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:23.187998 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-b45d9e9d-openshift-default-84948b9dcb-w47px"] Jul 28 22:41:24.405192 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:24.405159 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26f691f0-3715-4758-9928-2cf425ae448f" path="/var/lib/kubelet/pods/26f691f0-3715-4758-9928-2cf425ae448f/volumes" Jul 28 22:41:36.823469 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:36.823432 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml"] Jul 28 22:41:36.823903 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:36.823816 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75f8991f-441b-462a-a1a5-377b9134774e" containerName="maas-api" Jul 28 22:41:36.823903 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:36.823827 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="75f8991f-441b-462a-a1a5-377b9134774e" containerName="maas-api" Jul 28 22:41:36.823903 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:36.823837 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26f691f0-3715-4758-9928-2cf425ae448f" containerName="istio-proxy" Jul 28 22:41:36.823903 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:36.823842 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="26f691f0-3715-4758-9928-2cf425ae448f" containerName="istio-proxy" Jul 28 22:41:36.824060 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:36.823925 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="26f691f0-3715-4758-9928-2cf425ae448f" containerName="istio-proxy" Jul 28 22:41:36.824060 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:36.823933 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="26f691f0-3715-4758-9928-2cf425ae448f" containerName="istio-proxy" Jul 28 22:41:36.824060 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:36.823941 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="75f8991f-441b-462a-a1a5-377b9134774e" containerName="maas-api" Jul 28 22:41:36.828010 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:36.827993 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml" Jul 28 22:41:36.830253 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:36.830235 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-8f322d7b-openshift-default-dockercfg-7hh5m\"" Jul 28 22:41:36.836625 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:36.836601 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml"] Jul 28 22:41:36.913856 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:36.913818 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tknll\" (UniqueName: \"kubernetes.io/projected/7964a527-b6ab-4f88-b67e-7e286297aa41-kube-api-access-tknll\") pod \"e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml\" (UID: \"7964a527-b6ab-4f88-b67e-7e286297aa41\") " pod="openshift-ingress/e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml" Jul 28 22:41:36.914053 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:36.913873 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7964a527-b6ab-4f88-b67e-7e286297aa41-istio-podinfo\") pod \"e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml\" (UID: \"7964a527-b6ab-4f88-b67e-7e286297aa41\") " pod="openshift-ingress/e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml" Jul 28 22:41:36.914053 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:36.913894 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7964a527-b6ab-4f88-b67e-7e286297aa41-istio-envoy\") pod \"e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml\" (UID: \"7964a527-b6ab-4f88-b67e-7e286297aa41\") " pod="openshift-ingress/e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml" Jul 28 22:41:36.914053 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:36.913957 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7964a527-b6ab-4f88-b67e-7e286297aa41-credential-socket\") pod \"e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml\" (UID: \"7964a527-b6ab-4f88-b67e-7e286297aa41\") " pod="openshift-ingress/e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml" Jul 28 22:41:36.914053 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:36.914039 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7964a527-b6ab-4f88-b67e-7e286297aa41-workload-socket\") pod \"e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml\" (UID: \"7964a527-b6ab-4f88-b67e-7e286297aa41\") " pod="openshift-ingress/e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml" Jul 28 22:41:36.914229 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:36.914121 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7964a527-b6ab-4f88-b67e-7e286297aa41-workload-certs\") pod \"e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml\" (UID: \"7964a527-b6ab-4f88-b67e-7e286297aa41\") " pod="openshift-ingress/e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml" Jul 28 22:41:36.914229 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:36.914178 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7964a527-b6ab-4f88-b67e-7e286297aa41-istio-token\") pod \"e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml\" (UID: \"7964a527-b6ab-4f88-b67e-7e286297aa41\") " pod="openshift-ingress/e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml" Jul 28 22:41:36.914229 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:36.914202 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7964a527-b6ab-4f88-b67e-7e286297aa41-istiod-ca-cert\") pod \"e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml\" (UID: \"7964a527-b6ab-4f88-b67e-7e286297aa41\") " pod="openshift-ingress/e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml" Jul 28 22:41:36.914341 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:36.914227 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7964a527-b6ab-4f88-b67e-7e286297aa41-istio-data\") pod \"e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml\" (UID: \"7964a527-b6ab-4f88-b67e-7e286297aa41\") " pod="openshift-ingress/e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml" Jul 28 22:41:37.015498 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:37.015463 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7964a527-b6ab-4f88-b67e-7e286297aa41-istio-token\") pod \"e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml\" (UID: \"7964a527-b6ab-4f88-b67e-7e286297aa41\") " pod="openshift-ingress/e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml" Jul 28 22:41:37.015498 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:37.015498 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7964a527-b6ab-4f88-b67e-7e286297aa41-istiod-ca-cert\") pod \"e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml\" (UID: \"7964a527-b6ab-4f88-b67e-7e286297aa41\") " pod="openshift-ingress/e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml" Jul 28 22:41:37.015798 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:37.015517 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7964a527-b6ab-4f88-b67e-7e286297aa41-istio-data\") pod \"e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml\" (UID: \"7964a527-b6ab-4f88-b67e-7e286297aa41\") " pod="openshift-ingress/e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml" Jul 28 22:41:37.015798 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:37.015548 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tknll\" (UniqueName: \"kubernetes.io/projected/7964a527-b6ab-4f88-b67e-7e286297aa41-kube-api-access-tknll\") pod \"e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml\" (UID: \"7964a527-b6ab-4f88-b67e-7e286297aa41\") " pod="openshift-ingress/e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml" Jul 28 22:41:37.015798 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:37.015587 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7964a527-b6ab-4f88-b67e-7e286297aa41-istio-podinfo\") pod \"e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml\" (UID: \"7964a527-b6ab-4f88-b67e-7e286297aa41\") " pod="openshift-ingress/e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml" Jul 28 22:41:37.015798 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:37.015615 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7964a527-b6ab-4f88-b67e-7e286297aa41-istio-envoy\") pod \"e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml\" (UID: \"7964a527-b6ab-4f88-b67e-7e286297aa41\") " pod="openshift-ingress/e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml" Jul 28 22:41:37.015798 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:37.015652 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7964a527-b6ab-4f88-b67e-7e286297aa41-credential-socket\") pod \"e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml\" (UID: \"7964a527-b6ab-4f88-b67e-7e286297aa41\") " pod="openshift-ingress/e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml" Jul 28 22:41:37.015798 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:37.015687 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7964a527-b6ab-4f88-b67e-7e286297aa41-workload-socket\") pod \"e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml\" (UID: \"7964a527-b6ab-4f88-b67e-7e286297aa41\") " pod="openshift-ingress/e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml" Jul 28 22:41:37.015798 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:37.015769 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7964a527-b6ab-4f88-b67e-7e286297aa41-workload-certs\") pod \"e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml\" (UID: \"7964a527-b6ab-4f88-b67e-7e286297aa41\") " pod="openshift-ingress/e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml" Jul 28 22:41:37.016163 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:37.015977 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7964a527-b6ab-4f88-b67e-7e286297aa41-istio-data\") pod \"e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml\" (UID: \"7964a527-b6ab-4f88-b67e-7e286297aa41\") " pod="openshift-ingress/e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml" Jul 28 22:41:37.016163 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:37.016118 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7964a527-b6ab-4f88-b67e-7e286297aa41-credential-socket\") pod \"e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml\" (UID: \"7964a527-b6ab-4f88-b67e-7e286297aa41\") " pod="openshift-ingress/e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml" Jul 28 22:41:37.016275 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:37.016221 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7964a527-b6ab-4f88-b67e-7e286297aa41-workload-socket\") pod \"e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml\" (UID: \"7964a527-b6ab-4f88-b67e-7e286297aa41\") " pod="openshift-ingress/e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml" Jul 28 22:41:37.016338 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:37.016275 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7964a527-b6ab-4f88-b67e-7e286297aa41-workload-certs\") pod \"e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml\" (UID: \"7964a527-b6ab-4f88-b67e-7e286297aa41\") " pod="openshift-ingress/e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml" Jul 28 22:41:37.016451 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:37.016429 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7964a527-b6ab-4f88-b67e-7e286297aa41-istiod-ca-cert\") pod \"e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml\" (UID: \"7964a527-b6ab-4f88-b67e-7e286297aa41\") " pod="openshift-ingress/e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml" Jul 28 22:41:37.018086 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:37.018066 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7964a527-b6ab-4f88-b67e-7e286297aa41-istio-envoy\") pod \"e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml\" (UID: \"7964a527-b6ab-4f88-b67e-7e286297aa41\") " pod="openshift-ingress/e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml" Jul 28 22:41:37.018204 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:37.018182 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7964a527-b6ab-4f88-b67e-7e286297aa41-istio-podinfo\") pod \"e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml\" (UID: \"7964a527-b6ab-4f88-b67e-7e286297aa41\") " pod="openshift-ingress/e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml" Jul 28 22:41:37.023043 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:37.023010 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7964a527-b6ab-4f88-b67e-7e286297aa41-istio-token\") pod \"e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml\" (UID: \"7964a527-b6ab-4f88-b67e-7e286297aa41\") " pod="openshift-ingress/e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml" Jul 28 22:41:37.023263 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:37.023243 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tknll\" (UniqueName: \"kubernetes.io/projected/7964a527-b6ab-4f88-b67e-7e286297aa41-kube-api-access-tknll\") pod \"e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml\" (UID: \"7964a527-b6ab-4f88-b67e-7e286297aa41\") " pod="openshift-ingress/e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml" Jul 28 22:41:37.140007 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:37.139934 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml" Jul 28 22:41:37.288331 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:37.288165 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml"] Jul 28 22:41:37.290757 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:41:37.290698 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7964a527_b6ab_4f88_b67e_7e286297aa41.slice/crio-be01f7e65d073b88e0d34b6cbcfb2f9bece19eb1613104c2a9c8183fb48eac79 WatchSource:0}: Error finding container be01f7e65d073b88e0d34b6cbcfb2f9bece19eb1613104c2a9c8183fb48eac79: Status 404 returned error can't find the container with id be01f7e65d073b88e0d34b6cbcfb2f9bece19eb1613104c2a9c8183fb48eac79 Jul 28 22:41:37.293114 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:37.293081 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:41:37.293241 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:37.293144 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:41:37.293241 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:37.293173 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:41:38.246847 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:38.246804 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml" event={"ID":"7964a527-b6ab-4f88-b67e-7e286297aa41","Type":"ContainerStarted","Data":"28aab7d02d9050428c57b4f6790b144109f58eacaf68c296b967008e55e2f7e7"} Jul 28 22:41:38.247327 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:38.246862 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml" event={"ID":"7964a527-b6ab-4f88-b67e-7e286297aa41","Type":"ContainerStarted","Data":"be01f7e65d073b88e0d34b6cbcfb2f9bece19eb1613104c2a9c8183fb48eac79"} Jul 28 22:41:38.266625 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:38.266567 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml" podStartSLOduration=2.266548738 podStartE2EDuration="2.266548738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:41:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:41:38.265645103 +0000 UTC m=+1918.352903171" watchObservedRunningTime="2026-07-28 22:41:38.266548738 +0000 UTC m=+1918.353806808" Jul 28 22:41:39.141018 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:39.140979 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml" Jul 28 22:41:39.146197 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:39.146174 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml" Jul 28 22:41:39.251763 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:39.251729 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml" Jul 28 22:41:39.252824 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:39.252803 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml" Jul 28 22:41:40.386736 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:40.386687 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8f322d7b-7f94d659b9-9khhd"] Jul 28 22:41:40.387177 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:40.387070 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26f691f0-3715-4758-9928-2cf425ae448f" containerName="istio-proxy" Jul 28 22:41:40.387177 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:40.387082 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="26f691f0-3715-4758-9928-2cf425ae448f" containerName="istio-proxy" Jul 28 22:41:40.390171 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:40.390150 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f322d7b-7f94d659b9-9khhd" Jul 28 22:41:40.392436 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:40.392416 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-8f322d7b-serving-cert\"" Jul 28 22:41:40.416189 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:40.416160 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8f322d7b-7f94d659b9-9khhd"] Jul 28 22:41:40.450147 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:40.450113 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdfaeb3d-fe22-47ef-a539-e65b8a63f280-maas-api-tls\") pod \"maas-api-e2e-mt-8f322d7b-7f94d659b9-9khhd\" (UID: \"fdfaeb3d-fe22-47ef-a539-e65b8a63f280\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f322d7b-7f94d659b9-9khhd" Jul 28 22:41:40.450364 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:40.450339 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vll64\" (UniqueName: \"kubernetes.io/projected/fdfaeb3d-fe22-47ef-a539-e65b8a63f280-kube-api-access-vll64\") pod \"maas-api-e2e-mt-8f322d7b-7f94d659b9-9khhd\" (UID: \"fdfaeb3d-fe22-47ef-a539-e65b8a63f280\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f322d7b-7f94d659b9-9khhd" Jul 28 22:41:40.450482 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:40.450466 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdfaeb3d-fe22-47ef-a539-e65b8a63f280-openshift-service-ca\") pod \"maas-api-e2e-mt-8f322d7b-7f94d659b9-9khhd\" (UID: \"fdfaeb3d-fe22-47ef-a539-e65b8a63f280\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f322d7b-7f94d659b9-9khhd" Jul 28 22:41:40.551562 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:40.551538 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vll64\" (UniqueName: \"kubernetes.io/projected/fdfaeb3d-fe22-47ef-a539-e65b8a63f280-kube-api-access-vll64\") pod \"maas-api-e2e-mt-8f322d7b-7f94d659b9-9khhd\" (UID: \"fdfaeb3d-fe22-47ef-a539-e65b8a63f280\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f322d7b-7f94d659b9-9khhd" Jul 28 22:41:40.551650 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:40.551591 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdfaeb3d-fe22-47ef-a539-e65b8a63f280-openshift-service-ca\") pod \"maas-api-e2e-mt-8f322d7b-7f94d659b9-9khhd\" (UID: \"fdfaeb3d-fe22-47ef-a539-e65b8a63f280\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f322d7b-7f94d659b9-9khhd" Jul 28 22:41:40.551696 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:40.551653 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdfaeb3d-fe22-47ef-a539-e65b8a63f280-maas-api-tls\") pod \"maas-api-e2e-mt-8f322d7b-7f94d659b9-9khhd\" (UID: \"fdfaeb3d-fe22-47ef-a539-e65b8a63f280\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f322d7b-7f94d659b9-9khhd" Jul 28 22:41:40.551825 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:41:40.551805 2553 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-8f322d7b-serving-cert: secret "maas-api-e2e-mt-8f322d7b-serving-cert" not found Jul 28 22:41:40.551897 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:41:40.551887 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdfaeb3d-fe22-47ef-a539-e65b8a63f280-maas-api-tls podName:fdfaeb3d-fe22-47ef-a539-e65b8a63f280 nodeName:}" failed. No retries permitted until 2026-07-28 22:41:41.051864931 +0000 UTC m=+1921.139122983 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fdfaeb3d-fe22-47ef-a539-e65b8a63f280-maas-api-tls") pod "maas-api-e2e-mt-8f322d7b-7f94d659b9-9khhd" (UID: "fdfaeb3d-fe22-47ef-a539-e65b8a63f280") : secret "maas-api-e2e-mt-8f322d7b-serving-cert" not found Jul 28 22:41:40.552158 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:40.552137 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdfaeb3d-fe22-47ef-a539-e65b8a63f280-openshift-service-ca\") pod \"maas-api-e2e-mt-8f322d7b-7f94d659b9-9khhd\" (UID: \"fdfaeb3d-fe22-47ef-a539-e65b8a63f280\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f322d7b-7f94d659b9-9khhd" Jul 28 22:41:40.560126 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:40.560110 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vll64\" (UniqueName: \"kubernetes.io/projected/fdfaeb3d-fe22-47ef-a539-e65b8a63f280-kube-api-access-vll64\") pod \"maas-api-e2e-mt-8f322d7b-7f94d659b9-9khhd\" (UID: \"fdfaeb3d-fe22-47ef-a539-e65b8a63f280\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f322d7b-7f94d659b9-9khhd" Jul 28 22:41:41.057390 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:41.057359 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdfaeb3d-fe22-47ef-a539-e65b8a63f280-maas-api-tls\") pod \"maas-api-e2e-mt-8f322d7b-7f94d659b9-9khhd\" (UID: \"fdfaeb3d-fe22-47ef-a539-e65b8a63f280\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f322d7b-7f94d659b9-9khhd" Jul 28 22:41:41.060106 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:41.060076 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdfaeb3d-fe22-47ef-a539-e65b8a63f280-maas-api-tls\") pod \"maas-api-e2e-mt-8f322d7b-7f94d659b9-9khhd\" (UID: \"fdfaeb3d-fe22-47ef-a539-e65b8a63f280\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f322d7b-7f94d659b9-9khhd" Jul 28 22:41:41.302423 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:41.302391 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f322d7b-7f94d659b9-9khhd" Jul 28 22:41:41.647107 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:41.647084 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8f322d7b-7f94d659b9-9khhd"] Jul 28 22:41:41.648164 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:41:41.648140 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdfaeb3d_fe22_47ef_a539_e65b8a63f280.slice/crio-d2bbae3d9cdcf50b1937fb71253c0ac9cc7ca6145d73ce42cfba17d60cc07b22 WatchSource:0}: Error finding container d2bbae3d9cdcf50b1937fb71253c0ac9cc7ca6145d73ce42cfba17d60cc07b22: Status 404 returned error can't find the container with id d2bbae3d9cdcf50b1937fb71253c0ac9cc7ca6145d73ce42cfba17d60cc07b22 Jul 28 22:41:41.649590 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:41.649574 2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 22:41:42.270289 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:42.270253 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f322d7b-7f94d659b9-9khhd" event={"ID":"fdfaeb3d-fe22-47ef-a539-e65b8a63f280","Type":"ContainerStarted","Data":"c6650d61ea07de185fc19d2dd04effe6e57b68ff5aa8a99f10b9352125bc84ed"} Jul 28 22:41:42.270468 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:42.270317 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f322d7b-7f94d659b9-9khhd" event={"ID":"fdfaeb3d-fe22-47ef-a539-e65b8a63f280","Type":"ContainerStarted","Data":"d2bbae3d9cdcf50b1937fb71253c0ac9cc7ca6145d73ce42cfba17d60cc07b22"} Jul 28 22:41:42.270468 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:42.270344 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f322d7b-7f94d659b9-9khhd" Jul 28 22:41:42.299321 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:42.299250 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f322d7b-7f94d659b9-9khhd" podStartSLOduration=1.882011625 podStartE2EDuration="2.299230602s" totalImagesPullingTime="417.218977ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:41:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:41:42.299129923 +0000 UTC m=+1922.386387993" watchObservedRunningTime="2026-07-28 22:41:42.299230602 +0000 UTC m=+1922.386488672" Jul 28 22:41:48.281770 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:48.281737 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f322d7b-7f94d659b9-9khhd" Jul 28 22:41:54.253682 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:54.253648 2553 patch_prober.go:28] interesting pod/e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 28 22:41:54.254128 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:54.253720 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml" podUID="7964a527-b6ab-4f88-b67e-7e286297aa41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 28 22:41:54.353303 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:54.353266 2553 generic.go:356] "Generic (PLEG): container finished" podID="7964a527-b6ab-4f88-b67e-7e286297aa41" containerID="28aab7d02d9050428c57b4f6790b144109f58eacaf68c296b967008e55e2f7e7" exitCode=0 Jul 28 22:41:54.353483 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:54.353339 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml" event={"ID":"7964a527-b6ab-4f88-b67e-7e286297aa41","Type":"ContainerDied","Data":"28aab7d02d9050428c57b4f6790b144109f58eacaf68c296b967008e55e2f7e7"} Jul 28 22:41:54.353779 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:54.353762 2553 scope.go:117] "RemoveContainer" containerID="28aab7d02d9050428c57b4f6790b144109f58eacaf68c296b967008e55e2f7e7" Jul 28 22:41:54.354801 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:54.354775 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:41:54.354882 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:54.354858 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:41:54.354919 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:54.354909 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:41:55.360980 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:55.360945 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml" event={"ID":"7964a527-b6ab-4f88-b67e-7e286297aa41","Type":"ContainerStarted","Data":"5c5e772d72a5b8595dfbbbabc36d7035ec16defa5f9e6277d2efeb3328880bd4"} Jul 28 22:41:56.140564 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:56.140531 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml" Jul 28 22:41:56.140839 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:56.140818 2553 patch_prober.go:28] interesting pod/e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 28 22:41:56.140913 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:56.140867 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml" podUID="7964a527-b6ab-4f88-b67e-7e286297aa41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 28 22:41:57.140759 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:57.140684 2553 patch_prober.go:28] interesting pod/e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 28 22:41:57.141259 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:57.140785 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml" podUID="7964a527-b6ab-4f88-b67e-7e286297aa41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 28 22:41:58.140558 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:58.140516 2553 patch_prober.go:28] interesting pod/e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 28 22:41:58.140799 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:58.140591 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml" podUID="7964a527-b6ab-4f88-b67e-7e286297aa41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 28 22:41:59.140513 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:59.140477 2553 patch_prober.go:28] interesting pod/e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 28 22:41:59.140697 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:41:59.140532 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml" podUID="7964a527-b6ab-4f88-b67e-7e286297aa41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 28 22:42:00.144796 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:42:00.144764 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml" Jul 28 22:42:00.145240 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:42:00.145096 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml" Jul 28 22:42:00.145931 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:42:00.145910 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml" Jul 28 22:42:49.888192 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:42:49.888153 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8f322d7b-8nq8r"] Jul 28 22:42:49.891819 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:42:49.891793 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8f322d7b-8nq8r" Jul 28 22:42:49.894991 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:42:49.894216 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qf46x\"" Jul 28 22:42:49.900727 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:42:49.898137 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a10b4b02-097f-4aaf-91f0-87348872bbbd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-8f322d7b-8nq8r\" (UID: \"a10b4b02-097f-4aaf-91f0-87348872bbbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8f322d7b-8nq8r" Jul 28 22:42:49.900727 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:42:49.898358 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8f322d7b-8nq8r"] Jul 28 22:42:49.999005 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:42:49.998962 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a10b4b02-097f-4aaf-91f0-87348872bbbd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-8f322d7b-8nq8r\" (UID: \"a10b4b02-097f-4aaf-91f0-87348872bbbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8f322d7b-8nq8r" Jul 28 22:42:49.999554 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:42:49.999532 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a10b4b02-097f-4aaf-91f0-87348872bbbd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-8f322d7b-8nq8r\" (UID: \"a10b4b02-097f-4aaf-91f0-87348872bbbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8f322d7b-8nq8r" Jul 28 22:42:50.212489 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:42:50.212466 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8f322d7b-8nq8r" Jul 28 22:42:50.357770 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:42:50.357674 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8f322d7b-8nq8r"] Jul 28 22:42:50.360837 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:42:50.360806 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda10b4b02_097f_4aaf_91f0_87348872bbbd.slice/crio-be63424682f56a670f9e8d59876b79dbdad7ffe52490cad1b4a05b076532c0c2 WatchSource:0}: Error finding container be63424682f56a670f9e8d59876b79dbdad7ffe52490cad1b4a05b076532c0c2: Status 404 returned error can't find the container with id be63424682f56a670f9e8d59876b79dbdad7ffe52490cad1b4a05b076532c0c2 Jul 28 22:42:50.673475 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:42:50.673441 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8f322d7b-8nq8r" event={"ID":"a10b4b02-097f-4aaf-91f0-87348872bbbd","Type":"ContainerStarted","Data":"d1dc36016beec6c171b5c47e3d7416102f7c365fd49cf6c14cb96fcb3fa2171c"} Jul 28 22:42:50.673652 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:42:50.673496 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8f322d7b-8nq8r" event={"ID":"a10b4b02-097f-4aaf-91f0-87348872bbbd","Type":"ContainerStarted","Data":"be63424682f56a670f9e8d59876b79dbdad7ffe52490cad1b4a05b076532c0c2"} Jul 28 22:42:50.688586 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:42:50.688532 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8f322d7b-8nq8r" podStartSLOduration=1.688515628 podStartE2EDuration="1.688515628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:42:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:42:50.685977053 +0000 UTC m=+1990.773235123" watchObservedRunningTime="2026-07-28 22:42:50.688515628 +0000 UTC m=+1990.775773699" Jul 28 22:42:51.680974 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:42:51.680942 2553 generic.go:356] "Generic (PLEG): container finished" podID="a10b4b02-097f-4aaf-91f0-87348872bbbd" containerID="d1dc36016beec6c171b5c47e3d7416102f7c365fd49cf6c14cb96fcb3fa2171c" exitCode=0 Jul 28 22:42:51.681379 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:42:51.680985 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8f322d7b-8nq8r" event={"ID":"a10b4b02-097f-4aaf-91f0-87348872bbbd","Type":"ContainerDied","Data":"d1dc36016beec6c171b5c47e3d7416102f7c365fd49cf6c14cb96fcb3fa2171c"} Jul 28 22:42:52.824868 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:42:52.824844 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8f322d7b-8nq8r" Jul 28 22:42:52.919046 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:42:52.919013 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a10b4b02-097f-4aaf-91f0-87348872bbbd-maas-api-service-ca\") pod \"a10b4b02-097f-4aaf-91f0-87348872bbbd\" (UID: \"a10b4b02-097f-4aaf-91f0-87348872bbbd\") " Jul 28 22:42:52.919439 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:42:52.919417 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a10b4b02-097f-4aaf-91f0-87348872bbbd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a10b4b02-097f-4aaf-91f0-87348872bbbd" (UID: "a10b4b02-097f-4aaf-91f0-87348872bbbd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 22:42:53.019825 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:42:53.019790 2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a10b4b02-097f-4aaf-91f0-87348872bbbd-maas-api-service-ca\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:42:53.692976 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:42:53.692943 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8f322d7b-8nq8r" Jul 28 22:42:53.693157 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:42:53.692980 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8f322d7b-8nq8r" event={"ID":"a10b4b02-097f-4aaf-91f0-87348872bbbd","Type":"ContainerDied","Data":"be63424682f56a670f9e8d59876b79dbdad7ffe52490cad1b4a05b076532c0c2"} Jul 28 22:42:53.693157 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:42:53.693030 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be63424682f56a670f9e8d59876b79dbdad7ffe52490cad1b4a05b076532c0c2" Jul 28 22:43:05.385186 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:43:05.385151 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8f322d7b-7f94d659b9-9khhd"] Jul 28 22:43:05.385822 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:43:05.385402 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f322d7b-7f94d659b9-9khhd" podUID="fdfaeb3d-fe22-47ef-a539-e65b8a63f280" containerName="maas-api" containerID="cri-o://c6650d61ea07de185fc19d2dd04effe6e57b68ff5aa8a99f10b9352125bc84ed" gracePeriod=30 Jul 28 22:43:05.585563 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:43:05.585499 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8f322d7b-8nq8r"] Jul 28 22:43:05.588770 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:43:05.588745 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8f322d7b-8nq8r"] Jul 28 22:43:05.655081 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:43:05.655054 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f322d7b-7f94d659b9-9khhd" Jul 28 22:43:05.715003 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:43:05.714973 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdfaeb3d-fe22-47ef-a539-e65b8a63f280-maas-api-tls\") pod \"fdfaeb3d-fe22-47ef-a539-e65b8a63f280\" (UID: \"fdfaeb3d-fe22-47ef-a539-e65b8a63f280\") " Jul 28 22:43:05.715193 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:43:05.715063 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdfaeb3d-fe22-47ef-a539-e65b8a63f280-openshift-service-ca\") pod \"fdfaeb3d-fe22-47ef-a539-e65b8a63f280\" (UID: \"fdfaeb3d-fe22-47ef-a539-e65b8a63f280\") " Jul 28 22:43:05.715193 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:43:05.715126 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vll64\" (UniqueName: \"kubernetes.io/projected/fdfaeb3d-fe22-47ef-a539-e65b8a63f280-kube-api-access-vll64\") pod \"fdfaeb3d-fe22-47ef-a539-e65b8a63f280\" (UID: \"fdfaeb3d-fe22-47ef-a539-e65b8a63f280\") " Jul 28 22:43:05.715463 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:43:05.715440 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdfaeb3d-fe22-47ef-a539-e65b8a63f280-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fdfaeb3d-fe22-47ef-a539-e65b8a63f280" (UID: "fdfaeb3d-fe22-47ef-a539-e65b8a63f280"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 22:43:05.717348 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:43:05.717323 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdfaeb3d-fe22-47ef-a539-e65b8a63f280-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fdfaeb3d-fe22-47ef-a539-e65b8a63f280" (UID: "fdfaeb3d-fe22-47ef-a539-e65b8a63f280"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 22:43:05.717348 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:43:05.717323 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdfaeb3d-fe22-47ef-a539-e65b8a63f280-kube-api-access-vll64" (OuterVolumeSpecName: "kube-api-access-vll64") pod "fdfaeb3d-fe22-47ef-a539-e65b8a63f280" (UID: "fdfaeb3d-fe22-47ef-a539-e65b8a63f280"). InnerVolumeSpecName "kube-api-access-vll64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:43:05.764946 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:43:05.764909 2553 generic.go:356] "Generic (PLEG): container finished" podID="fdfaeb3d-fe22-47ef-a539-e65b8a63f280" containerID="c6650d61ea07de185fc19d2dd04effe6e57b68ff5aa8a99f10b9352125bc84ed" exitCode=0 Jul 28 22:43:05.765129 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:43:05.764970 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f322d7b-7f94d659b9-9khhd" Jul 28 22:43:05.765129 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:43:05.764996 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f322d7b-7f94d659b9-9khhd" event={"ID":"fdfaeb3d-fe22-47ef-a539-e65b8a63f280","Type":"ContainerDied","Data":"c6650d61ea07de185fc19d2dd04effe6e57b68ff5aa8a99f10b9352125bc84ed"} Jul 28 22:43:05.765129 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:43:05.765081 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f322d7b-7f94d659b9-9khhd" event={"ID":"fdfaeb3d-fe22-47ef-a539-e65b8a63f280","Type":"ContainerDied","Data":"d2bbae3d9cdcf50b1937fb71253c0ac9cc7ca6145d73ce42cfba17d60cc07b22"} Jul 28 22:43:05.765129 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:43:05.765110 2553 scope.go:117] "RemoveContainer" containerID="c6650d61ea07de185fc19d2dd04effe6e57b68ff5aa8a99f10b9352125bc84ed" Jul 28 22:43:05.774815 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:43:05.774798 2553 scope.go:117] "RemoveContainer" containerID="c6650d61ea07de185fc19d2dd04effe6e57b68ff5aa8a99f10b9352125bc84ed" Jul 28 22:43:05.775084 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:43:05.775063 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6650d61ea07de185fc19d2dd04effe6e57b68ff5aa8a99f10b9352125bc84ed\": container with ID starting with c6650d61ea07de185fc19d2dd04effe6e57b68ff5aa8a99f10b9352125bc84ed not found: ID does not exist" containerID="c6650d61ea07de185fc19d2dd04effe6e57b68ff5aa8a99f10b9352125bc84ed" Jul 28 22:43:05.775134 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:43:05.775092 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6650d61ea07de185fc19d2dd04effe6e57b68ff5aa8a99f10b9352125bc84ed"} err="failed to get container status \"c6650d61ea07de185fc19d2dd04effe6e57b68ff5aa8a99f10b9352125bc84ed\": rpc error: code = NotFound desc = could not find container \"c6650d61ea07de185fc19d2dd04effe6e57b68ff5aa8a99f10b9352125bc84ed\": container with ID starting with c6650d61ea07de185fc19d2dd04effe6e57b68ff5aa8a99f10b9352125bc84ed not found: ID does not exist" Jul 28 22:43:05.786084 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:43:05.786060 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8f322d7b-7f94d659b9-9khhd"] Jul 28 22:43:05.792263 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:43:05.792240 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8f322d7b-7f94d659b9-9khhd"] Jul 28 22:43:05.816283 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:43:05.816257 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vll64\" (UniqueName: \"kubernetes.io/projected/fdfaeb3d-fe22-47ef-a539-e65b8a63f280-kube-api-access-vll64\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:43:05.816283 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:43:05.816280 2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdfaeb3d-fe22-47ef-a539-e65b8a63f280-maas-api-tls\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:43:05.816421 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:43:05.816290 2553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdfaeb3d-fe22-47ef-a539-e65b8a63f280-openshift-service-ca\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:43:06.405582 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:43:06.405547 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a10b4b02-097f-4aaf-91f0-87348872bbbd" path="/var/lib/kubelet/pods/a10b4b02-097f-4aaf-91f0-87348872bbbd/volumes" Jul 28 22:43:06.406023 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:43:06.406007 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdfaeb3d-fe22-47ef-a539-e65b8a63f280" path="/var/lib/kubelet/pods/fdfaeb3d-fe22-47ef-a539-e65b8a63f280/volumes" Jul 28 22:43:40.556755 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:43:40.556699 2553 scope.go:117] "RemoveContainer" containerID="4faa3cdcaa7a88bb60e9ddbf8b4a1d7af673690cbde55263ff2907109362c8c4" Jul 28 22:43:56.793732 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:43:56.793681 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml"] Jul 28 22:43:56.794208 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:43:56.793955 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml" podUID="7964a527-b6ab-4f88-b67e-7e286297aa41" containerName="istio-proxy" containerID="cri-o://5c5e772d72a5b8595dfbbbabc36d7035ec16defa5f9e6277d2efeb3328880bd4" gracePeriod=30 Jul 28 22:44:00.147623 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:00.147577 2553 patch_prober.go:28] interesting pod/e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 28 22:44:00.148020 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:00.147653 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml" podUID="7964a527-b6ab-4f88-b67e-7e286297aa41" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 22:44:02.049495 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:02.049468 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml" Jul 28 22:44:02.074348 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:02.074134 2553 generic.go:356] "Generic (PLEG): container finished" podID="7964a527-b6ab-4f88-b67e-7e286297aa41" containerID="5c5e772d72a5b8595dfbbbabc36d7035ec16defa5f9e6277d2efeb3328880bd4" exitCode=0 Jul 28 22:44:02.074348 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:02.074241 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml" Jul 28 22:44:02.074348 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:02.074263 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml" event={"ID":"7964a527-b6ab-4f88-b67e-7e286297aa41","Type":"ContainerDied","Data":"5c5e772d72a5b8595dfbbbabc36d7035ec16defa5f9e6277d2efeb3328880bd4"} Jul 28 22:44:02.074348 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:02.074335 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml" event={"ID":"7964a527-b6ab-4f88-b67e-7e286297aa41","Type":"ContainerDied","Data":"be01f7e65d073b88e0d34b6cbcfb2f9bece19eb1613104c2a9c8183fb48eac79"} Jul 28 22:44:02.074667 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:02.074366 2553 scope.go:117] "RemoveContainer" containerID="5c5e772d72a5b8595dfbbbabc36d7035ec16defa5f9e6277d2efeb3328880bd4" Jul 28 22:44:02.083877 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:02.083852 2553 scope.go:117] "RemoveContainer" containerID="28aab7d02d9050428c57b4f6790b144109f58eacaf68c296b967008e55e2f7e7" Jul 28 22:44:02.092550 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:02.092526 2553 scope.go:117] "RemoveContainer" containerID="5c5e772d72a5b8595dfbbbabc36d7035ec16defa5f9e6277d2efeb3328880bd4" Jul 28 22:44:02.092950 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:44:02.092921 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c5e772d72a5b8595dfbbbabc36d7035ec16defa5f9e6277d2efeb3328880bd4\": container with ID starting with 5c5e772d72a5b8595dfbbbabc36d7035ec16defa5f9e6277d2efeb3328880bd4 not found: ID does not exist" containerID="5c5e772d72a5b8595dfbbbabc36d7035ec16defa5f9e6277d2efeb3328880bd4" Jul 28 22:44:02.093015 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:02.092967 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c5e772d72a5b8595dfbbbabc36d7035ec16defa5f9e6277d2efeb3328880bd4"} err="failed to get container status \"5c5e772d72a5b8595dfbbbabc36d7035ec16defa5f9e6277d2efeb3328880bd4\": rpc error: code = NotFound desc = could not find container \"5c5e772d72a5b8595dfbbbabc36d7035ec16defa5f9e6277d2efeb3328880bd4\": container with ID starting with 5c5e772d72a5b8595dfbbbabc36d7035ec16defa5f9e6277d2efeb3328880bd4 not found: ID does not exist" Jul 28 22:44:02.093015 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:02.092994 2553 scope.go:117] "RemoveContainer" containerID="28aab7d02d9050428c57b4f6790b144109f58eacaf68c296b967008e55e2f7e7" Jul 28 22:44:02.093267 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:44:02.093245 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28aab7d02d9050428c57b4f6790b144109f58eacaf68c296b967008e55e2f7e7\": container with ID starting with 28aab7d02d9050428c57b4f6790b144109f58eacaf68c296b967008e55e2f7e7 not found: ID does not exist" containerID="28aab7d02d9050428c57b4f6790b144109f58eacaf68c296b967008e55e2f7e7" Jul 28 22:44:02.093354 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:02.093275 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28aab7d02d9050428c57b4f6790b144109f58eacaf68c296b967008e55e2f7e7"} err="failed to get container status \"28aab7d02d9050428c57b4f6790b144109f58eacaf68c296b967008e55e2f7e7\": rpc error: code = NotFound desc = could not find container \"28aab7d02d9050428c57b4f6790b144109f58eacaf68c296b967008e55e2f7e7\": container with ID starting with 28aab7d02d9050428c57b4f6790b144109f58eacaf68c296b967008e55e2f7e7 not found: ID does not exist" Jul 28 22:44:02.107750 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:02.107727 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7964a527-b6ab-4f88-b67e-7e286297aa41-istio-token\") pod \"7964a527-b6ab-4f88-b67e-7e286297aa41\" (UID: \"7964a527-b6ab-4f88-b67e-7e286297aa41\") " Jul 28 22:44:02.107861 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:02.107760 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tknll\" (UniqueName: \"kubernetes.io/projected/7964a527-b6ab-4f88-b67e-7e286297aa41-kube-api-access-tknll\") pod \"7964a527-b6ab-4f88-b67e-7e286297aa41\" (UID: \"7964a527-b6ab-4f88-b67e-7e286297aa41\") " Jul 28 22:44:02.107861 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:02.107794 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7964a527-b6ab-4f88-b67e-7e286297aa41-istio-envoy\") pod \"7964a527-b6ab-4f88-b67e-7e286297aa41\" (UID: \"7964a527-b6ab-4f88-b67e-7e286297aa41\") " Jul 28 22:44:02.107978 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:02.107958 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7964a527-b6ab-4f88-b67e-7e286297aa41-credential-socket\") pod \"7964a527-b6ab-4f88-b67e-7e286297aa41\" (UID: \"7964a527-b6ab-4f88-b67e-7e286297aa41\") " Jul 28 22:44:02.108032 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:02.108010 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7964a527-b6ab-4f88-b67e-7e286297aa41-workload-certs\") pod \"7964a527-b6ab-4f88-b67e-7e286297aa41\" (UID: \"7964a527-b6ab-4f88-b67e-7e286297aa41\") " Jul 28 22:44:02.108075 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:02.108034 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7964a527-b6ab-4f88-b67e-7e286297aa41-istio-data\") pod \"7964a527-b6ab-4f88-b67e-7e286297aa41\" (UID: \"7964a527-b6ab-4f88-b67e-7e286297aa41\") " Jul 28 22:44:02.108130 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:02.108086 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7964a527-b6ab-4f88-b67e-7e286297aa41-istio-podinfo\") pod \"7964a527-b6ab-4f88-b67e-7e286297aa41\" (UID: \"7964a527-b6ab-4f88-b67e-7e286297aa41\") " Jul 28 22:44:02.108130 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:02.108115 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7964a527-b6ab-4f88-b67e-7e286297aa41-workload-socket\") pod \"7964a527-b6ab-4f88-b67e-7e286297aa41\" (UID: \"7964a527-b6ab-4f88-b67e-7e286297aa41\") " Jul 28 22:44:02.108238 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:02.108166 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7964a527-b6ab-4f88-b67e-7e286297aa41-istiod-ca-cert\") pod \"7964a527-b6ab-4f88-b67e-7e286297aa41\" (UID: \"7964a527-b6ab-4f88-b67e-7e286297aa41\") " Jul 28 22:44:02.108313 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:02.108286 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7964a527-b6ab-4f88-b67e-7e286297aa41-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7964a527-b6ab-4f88-b67e-7e286297aa41" (UID: "7964a527-b6ab-4f88-b67e-7e286297aa41"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:44:02.108367 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:02.108304 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7964a527-b6ab-4f88-b67e-7e286297aa41-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7964a527-b6ab-4f88-b67e-7e286297aa41" (UID: "7964a527-b6ab-4f88-b67e-7e286297aa41"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:44:02.108526 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:02.108508 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7964a527-b6ab-4f88-b67e-7e286297aa41-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7964a527-b6ab-4f88-b67e-7e286297aa41" (UID: "7964a527-b6ab-4f88-b67e-7e286297aa41"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:44:02.108628 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:02.108613 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7964a527-b6ab-4f88-b67e-7e286297aa41-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7964a527-b6ab-4f88-b67e-7e286297aa41" (UID: "7964a527-b6ab-4f88-b67e-7e286297aa41"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 22:44:02.108695 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:02.108588 2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7964a527-b6ab-4f88-b67e-7e286297aa41-credential-socket\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:44:02.108779 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:02.108722 2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7964a527-b6ab-4f88-b67e-7e286297aa41-workload-certs\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:44:02.108952 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:02.108927 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7964a527-b6ab-4f88-b67e-7e286297aa41-istio-data" (OuterVolumeSpecName: "istio-data") pod "7964a527-b6ab-4f88-b67e-7e286297aa41" (UID: "7964a527-b6ab-4f88-b67e-7e286297aa41"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:44:02.110189 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:02.110160 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7964a527-b6ab-4f88-b67e-7e286297aa41-istio-token" (OuterVolumeSpecName: "istio-token") pod "7964a527-b6ab-4f88-b67e-7e286297aa41" (UID: "7964a527-b6ab-4f88-b67e-7e286297aa41"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:44:02.110615 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:02.110593 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7964a527-b6ab-4f88-b67e-7e286297aa41-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7964a527-b6ab-4f88-b67e-7e286297aa41" (UID: "7964a527-b6ab-4f88-b67e-7e286297aa41"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 22:44:02.110747 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:02.110729 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7964a527-b6ab-4f88-b67e-7e286297aa41-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7964a527-b6ab-4f88-b67e-7e286297aa41" (UID: "7964a527-b6ab-4f88-b67e-7e286297aa41"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:44:02.110812 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:02.110786 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7964a527-b6ab-4f88-b67e-7e286297aa41-kube-api-access-tknll" (OuterVolumeSpecName: "kube-api-access-tknll") pod "7964a527-b6ab-4f88-b67e-7e286297aa41" (UID: "7964a527-b6ab-4f88-b67e-7e286297aa41"). InnerVolumeSpecName "kube-api-access-tknll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:44:02.209416 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:02.209384 2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7964a527-b6ab-4f88-b67e-7e286297aa41-istiod-ca-cert\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:44:02.209416 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:02.209415 2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7964a527-b6ab-4f88-b67e-7e286297aa41-istio-token\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:44:02.209627 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:02.209431 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tknll\" (UniqueName: \"kubernetes.io/projected/7964a527-b6ab-4f88-b67e-7e286297aa41-kube-api-access-tknll\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:44:02.209627 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:02.209441 2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7964a527-b6ab-4f88-b67e-7e286297aa41-istio-envoy\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:44:02.209627 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:02.209450 2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7964a527-b6ab-4f88-b67e-7e286297aa41-istio-data\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:44:02.209627 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:02.209457 2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7964a527-b6ab-4f88-b67e-7e286297aa41-istio-podinfo\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:44:02.209627 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:02.209464 2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7964a527-b6ab-4f88-b67e-7e286297aa41-workload-socket\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:44:02.397169 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:02.397133 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml"] Jul 28 22:44:02.409992 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:02.409960 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-8f322d7b-openshift-default-5976c684cb-zvjml"] Jul 28 22:44:04.405388 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:04.405354 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7964a527-b6ab-4f88-b67e-7e286297aa41" path="/var/lib/kubelet/pods/7964a527-b6ab-4f88-b67e-7e286297aa41/volumes" Jul 28 22:44:40.490315 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:40.490278 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mldhp_e1ced996-f136-4d3e-9c85-a5b7809b4517/console-operator/1.log" Jul 28 22:44:40.495439 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:40.495418 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mldhp_e1ced996-f136-4d3e-9c85-a5b7809b4517/console-operator/1.log" Jul 28 22:44:40.581126 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:40.581094 2553 scope.go:117] "RemoveContainer" containerID="000772f6a0d14c1eb05391df376d2f263786cf9d7b3bd4c9cbe8fbbff888d527" Jul 28 22:44:52.239464 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:52.239428 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-104d62-openshift-default-7647657476-hbjqb"] Jul 28 22:44:52.239870 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:52.239786 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7964a527-b6ab-4f88-b67e-7e286297aa41" containerName="istio-proxy" Jul 28 22:44:52.239870 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:52.239797 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="7964a527-b6ab-4f88-b67e-7e286297aa41" containerName="istio-proxy" Jul 28 22:44:52.239870 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:52.239808 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7964a527-b6ab-4f88-b67e-7e286297aa41" containerName="istio-proxy" Jul 28 22:44:52.239870 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:52.239814 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="7964a527-b6ab-4f88-b67e-7e286297aa41" containerName="istio-proxy" Jul 28 22:44:52.239870 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:52.239825 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a10b4b02-097f-4aaf-91f0-87348872bbbd" containerName="revoke-keys" Jul 28 22:44:52.239870 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:52.239830 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="a10b4b02-097f-4aaf-91f0-87348872bbbd" containerName="revoke-keys" Jul 28 22:44:52.239870 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:52.239838 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdfaeb3d-fe22-47ef-a539-e65b8a63f280" containerName="maas-api" Jul 28 22:44:52.239870 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:52.239843 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdfaeb3d-fe22-47ef-a539-e65b8a63f280" containerName="maas-api" Jul 28 22:44:52.240133 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:52.239897 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="a10b4b02-097f-4aaf-91f0-87348872bbbd" containerName="revoke-keys" Jul 28 22:44:52.240133 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:52.239906 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="7964a527-b6ab-4f88-b67e-7e286297aa41" containerName="istio-proxy" Jul 28 22:44:52.240133 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:52.239914 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdfaeb3d-fe22-47ef-a539-e65b8a63f280" containerName="maas-api" Jul 28 22:44:52.242794 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:52.242775 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-104d62-openshift-default-7647657476-hbjqb" Jul 28 22:44:52.245080 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:52.245058 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-104d62-openshift-default-dockercfg-q2lk6\"" Jul 28 22:44:52.256151 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:52.256124 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-104d62-openshift-default-7647657476-hbjqb"] Jul 28 22:44:52.344800 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:52.344771 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89223ac8-2798-48fe-b8c8-5e255b7b0bf2-credential-socket\") pod \"e2e-inf-a-104d62-openshift-default-7647657476-hbjqb\" (UID: \"89223ac8-2798-48fe-b8c8-5e255b7b0bf2\") " pod="openshift-ingress/e2e-inf-a-104d62-openshift-default-7647657476-hbjqb" Jul 28 22:44:52.344985 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:52.344823 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89223ac8-2798-48fe-b8c8-5e255b7b0bf2-istio-data\") pod \"e2e-inf-a-104d62-openshift-default-7647657476-hbjqb\" (UID: \"89223ac8-2798-48fe-b8c8-5e255b7b0bf2\") " pod="openshift-ingress/e2e-inf-a-104d62-openshift-default-7647657476-hbjqb" Jul 28 22:44:52.344985 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:52.344885 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gwhq\" (UniqueName: \"kubernetes.io/projected/89223ac8-2798-48fe-b8c8-5e255b7b0bf2-kube-api-access-9gwhq\") pod \"e2e-inf-a-104d62-openshift-default-7647657476-hbjqb\" (UID: \"89223ac8-2798-48fe-b8c8-5e255b7b0bf2\") " pod="openshift-ingress/e2e-inf-a-104d62-openshift-default-7647657476-hbjqb" Jul 28 22:44:52.344985 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:52.344937 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89223ac8-2798-48fe-b8c8-5e255b7b0bf2-workload-socket\") pod \"e2e-inf-a-104d62-openshift-default-7647657476-hbjqb\" (UID: \"89223ac8-2798-48fe-b8c8-5e255b7b0bf2\") " pod="openshift-ingress/e2e-inf-a-104d62-openshift-default-7647657476-hbjqb" Jul 28 22:44:52.344985 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:52.344955 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89223ac8-2798-48fe-b8c8-5e255b7b0bf2-workload-certs\") pod \"e2e-inf-a-104d62-openshift-default-7647657476-hbjqb\" (UID: \"89223ac8-2798-48fe-b8c8-5e255b7b0bf2\") " pod="openshift-ingress/e2e-inf-a-104d62-openshift-default-7647657476-hbjqb" Jul 28 22:44:52.345129 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:52.344989 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89223ac8-2798-48fe-b8c8-5e255b7b0bf2-istio-token\") pod \"e2e-inf-a-104d62-openshift-default-7647657476-hbjqb\" (UID: \"89223ac8-2798-48fe-b8c8-5e255b7b0bf2\") " pod="openshift-ingress/e2e-inf-a-104d62-openshift-default-7647657476-hbjqb" Jul 28 22:44:52.345129 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:52.345007 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89223ac8-2798-48fe-b8c8-5e255b7b0bf2-istiod-ca-cert\") pod \"e2e-inf-a-104d62-openshift-default-7647657476-hbjqb\" (UID: \"89223ac8-2798-48fe-b8c8-5e255b7b0bf2\") " pod="openshift-ingress/e2e-inf-a-104d62-openshift-default-7647657476-hbjqb" Jul 28 22:44:52.345129 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:52.345024 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89223ac8-2798-48fe-b8c8-5e255b7b0bf2-istio-envoy\") pod \"e2e-inf-a-104d62-openshift-default-7647657476-hbjqb\" (UID: \"89223ac8-2798-48fe-b8c8-5e255b7b0bf2\") " pod="openshift-ingress/e2e-inf-a-104d62-openshift-default-7647657476-hbjqb" Jul 28 22:44:52.345129 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:52.345100 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89223ac8-2798-48fe-b8c8-5e255b7b0bf2-istio-podinfo\") pod \"e2e-inf-a-104d62-openshift-default-7647657476-hbjqb\" (UID: \"89223ac8-2798-48fe-b8c8-5e255b7b0bf2\") " pod="openshift-ingress/e2e-inf-a-104d62-openshift-default-7647657476-hbjqb" Jul 28 22:44:52.445947 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:52.445912 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89223ac8-2798-48fe-b8c8-5e255b7b0bf2-istio-podinfo\") pod \"e2e-inf-a-104d62-openshift-default-7647657476-hbjqb\" (UID: \"89223ac8-2798-48fe-b8c8-5e255b7b0bf2\") " pod="openshift-ingress/e2e-inf-a-104d62-openshift-default-7647657476-hbjqb" Jul 28 22:44:52.446168 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:52.445999 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89223ac8-2798-48fe-b8c8-5e255b7b0bf2-credential-socket\") pod \"e2e-inf-a-104d62-openshift-default-7647657476-hbjqb\" (UID: \"89223ac8-2798-48fe-b8c8-5e255b7b0bf2\") " pod="openshift-ingress/e2e-inf-a-104d62-openshift-default-7647657476-hbjqb" Jul 28 22:44:52.446240 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:52.446184 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89223ac8-2798-48fe-b8c8-5e255b7b0bf2-istio-data\") pod \"e2e-inf-a-104d62-openshift-default-7647657476-hbjqb\" (UID: \"89223ac8-2798-48fe-b8c8-5e255b7b0bf2\") " pod="openshift-ingress/e2e-inf-a-104d62-openshift-default-7647657476-hbjqb" Jul 28 22:44:52.446299 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:52.446242 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gwhq\" (UniqueName: \"kubernetes.io/projected/89223ac8-2798-48fe-b8c8-5e255b7b0bf2-kube-api-access-9gwhq\") pod \"e2e-inf-a-104d62-openshift-default-7647657476-hbjqb\" (UID: \"89223ac8-2798-48fe-b8c8-5e255b7b0bf2\") " pod="openshift-ingress/e2e-inf-a-104d62-openshift-default-7647657476-hbjqb" Jul 28 22:44:52.446355 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:52.446323 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89223ac8-2798-48fe-b8c8-5e255b7b0bf2-workload-socket\") pod \"e2e-inf-a-104d62-openshift-default-7647657476-hbjqb\" (UID: \"89223ac8-2798-48fe-b8c8-5e255b7b0bf2\") " pod="openshift-ingress/e2e-inf-a-104d62-openshift-default-7647657476-hbjqb" Jul 28 22:44:52.446494 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:52.446474 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89223ac8-2798-48fe-b8c8-5e255b7b0bf2-workload-certs\") pod \"e2e-inf-a-104d62-openshift-default-7647657476-hbjqb\" (UID: \"89223ac8-2798-48fe-b8c8-5e255b7b0bf2\") " pod="openshift-ingress/e2e-inf-a-104d62-openshift-default-7647657476-hbjqb" Jul 28 22:44:52.446550 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:52.446531 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89223ac8-2798-48fe-b8c8-5e255b7b0bf2-istio-data\") pod \"e2e-inf-a-104d62-openshift-default-7647657476-hbjqb\" (UID: \"89223ac8-2798-48fe-b8c8-5e255b7b0bf2\") " pod="openshift-ingress/e2e-inf-a-104d62-openshift-default-7647657476-hbjqb" Jul 28 22:44:52.446602 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:52.446545 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89223ac8-2798-48fe-b8c8-5e255b7b0bf2-istio-token\") pod \"e2e-inf-a-104d62-openshift-default-7647657476-hbjqb\" (UID: \"89223ac8-2798-48fe-b8c8-5e255b7b0bf2\") " pod="openshift-ingress/e2e-inf-a-104d62-openshift-default-7647657476-hbjqb" Jul 28 22:44:52.446602 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:52.446477 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89223ac8-2798-48fe-b8c8-5e255b7b0bf2-credential-socket\") pod \"e2e-inf-a-104d62-openshift-default-7647657476-hbjqb\" (UID: \"89223ac8-2798-48fe-b8c8-5e255b7b0bf2\") " pod="openshift-ingress/e2e-inf-a-104d62-openshift-default-7647657476-hbjqb" Jul 28 22:44:52.446602 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:52.446570 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89223ac8-2798-48fe-b8c8-5e255b7b0bf2-istiod-ca-cert\") pod \"e2e-inf-a-104d62-openshift-default-7647657476-hbjqb\" (UID: \"89223ac8-2798-48fe-b8c8-5e255b7b0bf2\") " pod="openshift-ingress/e2e-inf-a-104d62-openshift-default-7647657476-hbjqb" Jul 28 22:44:52.446788 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:52.446612 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89223ac8-2798-48fe-b8c8-5e255b7b0bf2-istio-envoy\") pod \"e2e-inf-a-104d62-openshift-default-7647657476-hbjqb\" (UID: \"89223ac8-2798-48fe-b8c8-5e255b7b0bf2\") " pod="openshift-ingress/e2e-inf-a-104d62-openshift-default-7647657476-hbjqb" Jul 28 22:44:52.446788 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:52.446655 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89223ac8-2798-48fe-b8c8-5e255b7b0bf2-workload-socket\") pod \"e2e-inf-a-104d62-openshift-default-7647657476-hbjqb\" (UID: \"89223ac8-2798-48fe-b8c8-5e255b7b0bf2\") " pod="openshift-ingress/e2e-inf-a-104d62-openshift-default-7647657476-hbjqb" Jul 28 22:44:52.446788 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:52.446738 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89223ac8-2798-48fe-b8c8-5e255b7b0bf2-workload-certs\") pod \"e2e-inf-a-104d62-openshift-default-7647657476-hbjqb\" (UID: \"89223ac8-2798-48fe-b8c8-5e255b7b0bf2\") " pod="openshift-ingress/e2e-inf-a-104d62-openshift-default-7647657476-hbjqb" Jul 28 22:44:52.447201 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:52.447176 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89223ac8-2798-48fe-b8c8-5e255b7b0bf2-istiod-ca-cert\") pod \"e2e-inf-a-104d62-openshift-default-7647657476-hbjqb\" (UID: \"89223ac8-2798-48fe-b8c8-5e255b7b0bf2\") " pod="openshift-ingress/e2e-inf-a-104d62-openshift-default-7647657476-hbjqb" Jul 28 22:44:52.448907 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:52.448885 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89223ac8-2798-48fe-b8c8-5e255b7b0bf2-istio-envoy\") pod \"e2e-inf-a-104d62-openshift-default-7647657476-hbjqb\" (UID: \"89223ac8-2798-48fe-b8c8-5e255b7b0bf2\") " pod="openshift-ingress/e2e-inf-a-104d62-openshift-default-7647657476-hbjqb" Jul 28 22:44:52.449125 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:52.449106 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89223ac8-2798-48fe-b8c8-5e255b7b0bf2-istio-podinfo\") pod \"e2e-inf-a-104d62-openshift-default-7647657476-hbjqb\" (UID: \"89223ac8-2798-48fe-b8c8-5e255b7b0bf2\") " pod="openshift-ingress/e2e-inf-a-104d62-openshift-default-7647657476-hbjqb" Jul 28 22:44:52.454048 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:52.454021 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89223ac8-2798-48fe-b8c8-5e255b7b0bf2-istio-token\") pod \"e2e-inf-a-104d62-openshift-default-7647657476-hbjqb\" (UID: \"89223ac8-2798-48fe-b8c8-5e255b7b0bf2\") " pod="openshift-ingress/e2e-inf-a-104d62-openshift-default-7647657476-hbjqb" Jul 28 22:44:52.454154 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:52.454090 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gwhq\" (UniqueName: \"kubernetes.io/projected/89223ac8-2798-48fe-b8c8-5e255b7b0bf2-kube-api-access-9gwhq\") pod \"e2e-inf-a-104d62-openshift-default-7647657476-hbjqb\" (UID: \"89223ac8-2798-48fe-b8c8-5e255b7b0bf2\") " pod="openshift-ingress/e2e-inf-a-104d62-openshift-default-7647657476-hbjqb" Jul 28 22:44:52.553325 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:52.553234 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-104d62-openshift-default-7647657476-hbjqb" Jul 28 22:44:52.698648 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:52.698621 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-104d62-openshift-default-7647657476-hbjqb"] Jul 28 22:44:52.700599 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:44:52.700566 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89223ac8_2798_48fe_b8c8_5e255b7b0bf2.slice/crio-4656bc2ba7504983feb5a3ed8d76c4377ec73cf7da9fabd2757ee5c57b62f3d0 WatchSource:0}: Error finding container 4656bc2ba7504983feb5a3ed8d76c4377ec73cf7da9fabd2757ee5c57b62f3d0: Status 404 returned error can't find the container with id 4656bc2ba7504983feb5a3ed8d76c4377ec73cf7da9fabd2757ee5c57b62f3d0 Jul 28 22:44:52.702873 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:52.702842 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:44:52.702958 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:52.702906 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:44:52.702958 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:52.702934 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:44:53.348718 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:53.348677 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-104d62-openshift-default-7647657476-hbjqb" event={"ID":"89223ac8-2798-48fe-b8c8-5e255b7b0bf2","Type":"ContainerStarted","Data":"afd79614dbb0638e66e48f03f53069b5f547839498b30a9aff53a66c6b1821b9"} Jul 28 22:44:53.349230 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:53.348743 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-104d62-openshift-default-7647657476-hbjqb" event={"ID":"89223ac8-2798-48fe-b8c8-5e255b7b0bf2","Type":"ContainerStarted","Data":"4656bc2ba7504983feb5a3ed8d76c4377ec73cf7da9fabd2757ee5c57b62f3d0"} Jul 28 22:44:53.368336 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:53.368280 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-104d62-openshift-default-7647657476-hbjqb" podStartSLOduration=1.368260508 podStartE2EDuration="1.368260508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:44:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:44:53.366202975 +0000 UTC m=+2113.453461048" watchObservedRunningTime="2026-07-28 22:44:53.368260508 +0000 UTC m=+2113.455518580" Jul 28 22:44:53.554370 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:53.554331 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-104d62-openshift-default-7647657476-hbjqb" Jul 28 22:44:53.559619 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:53.559588 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-104d62-openshift-default-7647657476-hbjqb" Jul 28 22:44:54.354986 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:54.354953 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-104d62-openshift-default-7647657476-hbjqb" Jul 28 22:44:54.356009 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:54.355989 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-104d62-openshift-default-7647657476-hbjqb" Jul 28 22:44:55.436532 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:55.436502 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-104d62-5df9b68c94-dxqjf"] Jul 28 22:44:55.436942 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:55.436928 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="7964a527-b6ab-4f88-b67e-7e286297aa41" containerName="istio-proxy" Jul 28 22:44:55.440087 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:55.440067 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-104d62-5df9b68c94-dxqjf" Jul 28 22:44:55.442437 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:55.442409 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-104d62-serving-cert\"" Jul 28 22:44:55.452242 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:55.452222 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-104d62-5df9b68c94-dxqjf"] Jul 28 22:44:55.475474 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:55.475443 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pklx\" (UniqueName: \"kubernetes.io/projected/83e80fde-c25b-410e-9523-281a4a0450c5-kube-api-access-9pklx\") pod \"maas-api-e2e-inf-a-104d62-5df9b68c94-dxqjf\" (UID: \"83e80fde-c25b-410e-9523-281a4a0450c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-104d62-5df9b68c94-dxqjf" Jul 28 22:44:55.475642 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:55.475483 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83e80fde-c25b-410e-9523-281a4a0450c5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-104d62-5df9b68c94-dxqjf\" (UID: \"83e80fde-c25b-410e-9523-281a4a0450c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-104d62-5df9b68c94-dxqjf" Jul 28 22:44:55.475642 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:55.475548 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83e80fde-c25b-410e-9523-281a4a0450c5-maas-api-tls\") pod \"maas-api-e2e-inf-a-104d62-5df9b68c94-dxqjf\" (UID: \"83e80fde-c25b-410e-9523-281a4a0450c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-104d62-5df9b68c94-dxqjf" Jul 28 22:44:55.577032 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:55.576998 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83e80fde-c25b-410e-9523-281a4a0450c5-maas-api-tls\") pod \"maas-api-e2e-inf-a-104d62-5df9b68c94-dxqjf\" (UID: \"83e80fde-c25b-410e-9523-281a4a0450c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-104d62-5df9b68c94-dxqjf" Jul 28 22:44:55.577213 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:55.577090 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pklx\" (UniqueName: \"kubernetes.io/projected/83e80fde-c25b-410e-9523-281a4a0450c5-kube-api-access-9pklx\") pod \"maas-api-e2e-inf-a-104d62-5df9b68c94-dxqjf\" (UID: \"83e80fde-c25b-410e-9523-281a4a0450c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-104d62-5df9b68c94-dxqjf" Jul 28 22:44:55.577213 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:55.577117 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83e80fde-c25b-410e-9523-281a4a0450c5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-104d62-5df9b68c94-dxqjf\" (UID: \"83e80fde-c25b-410e-9523-281a4a0450c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-104d62-5df9b68c94-dxqjf" Jul 28 22:44:55.577213 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:44:55.577157 2553 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-104d62-serving-cert: secret "maas-api-e2e-inf-a-104d62-serving-cert" not found Jul 28 22:44:55.577337 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:44:55.577222 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83e80fde-c25b-410e-9523-281a4a0450c5-maas-api-tls podName:83e80fde-c25b-410e-9523-281a4a0450c5 nodeName:}" failed. No retries permitted until 2026-07-28 22:44:56.077206014 +0000 UTC m=+2116.164464061 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/83e80fde-c25b-410e-9523-281a4a0450c5-maas-api-tls") pod "maas-api-e2e-inf-a-104d62-5df9b68c94-dxqjf" (UID: "83e80fde-c25b-410e-9523-281a4a0450c5") : secret "maas-api-e2e-inf-a-104d62-serving-cert" not found Jul 28 22:44:55.577660 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:55.577644 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83e80fde-c25b-410e-9523-281a4a0450c5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-104d62-5df9b68c94-dxqjf\" (UID: \"83e80fde-c25b-410e-9523-281a4a0450c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-104d62-5df9b68c94-dxqjf" Jul 28 22:44:55.585278 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:55.585255 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pklx\" (UniqueName: \"kubernetes.io/projected/83e80fde-c25b-410e-9523-281a4a0450c5-kube-api-access-9pklx\") pod \"maas-api-e2e-inf-a-104d62-5df9b68c94-dxqjf\" (UID: \"83e80fde-c25b-410e-9523-281a4a0450c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-104d62-5df9b68c94-dxqjf" Jul 28 22:44:56.082053 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:56.082024 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83e80fde-c25b-410e-9523-281a4a0450c5-maas-api-tls\") pod \"maas-api-e2e-inf-a-104d62-5df9b68c94-dxqjf\" (UID: \"83e80fde-c25b-410e-9523-281a4a0450c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-104d62-5df9b68c94-dxqjf" Jul 28 22:44:56.085209 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:56.085179 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83e80fde-c25b-410e-9523-281a4a0450c5-maas-api-tls\") pod \"maas-api-e2e-inf-a-104d62-5df9b68c94-dxqjf\" (UID: \"83e80fde-c25b-410e-9523-281a4a0450c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-104d62-5df9b68c94-dxqjf" Jul 28 22:44:56.352762 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:56.352644 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-104d62-5df9b68c94-dxqjf" Jul 28 22:44:56.500232 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:56.500207 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-104d62-5df9b68c94-dxqjf"] Jul 28 22:44:56.502165 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:44:56.502133 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83e80fde_c25b_410e_9523_281a4a0450c5.slice/crio-db36cc7819f1dfdc931bce02d804c8f758b189e3f79ab5d382437a00b8085ea0 WatchSource:0}: Error finding container db36cc7819f1dfdc931bce02d804c8f758b189e3f79ab5d382437a00b8085ea0: Status 404 returned error can't find the container with id db36cc7819f1dfdc931bce02d804c8f758b189e3f79ab5d382437a00b8085ea0 Jul 28 22:44:57.379396 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:57.379358 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-104d62-5df9b68c94-dxqjf" event={"ID":"83e80fde-c25b-410e-9523-281a4a0450c5","Type":"ContainerStarted","Data":"7bef1945abb3bdcd1a15fa4fba07a410f64a52f0da4fc2f630dae93484900104"} Jul 28 22:44:57.379558 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:57.379421 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-104d62-5df9b68c94-dxqjf" Jul 28 22:44:57.379558 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:57.379446 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-104d62-5df9b68c94-dxqjf" event={"ID":"83e80fde-c25b-410e-9523-281a4a0450c5","Type":"ContainerStarted","Data":"db36cc7819f1dfdc931bce02d804c8f758b189e3f79ab5d382437a00b8085ea0"} Jul 28 22:44:57.408132 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:44:57.408081 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-104d62-5df9b68c94-dxqjf" podStartSLOduration=1.94939017 podStartE2EDuration="2.408062554s" totalImagesPullingTime="458.672384ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:44:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:44:57.404808399 +0000 UTC m=+2117.492066467" watchObservedRunningTime="2026-07-28 22:44:57.408062554 +0000 UTC m=+2117.495320624" Jul 28 22:45:03.398205 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:03.398175 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-104d62-5df9b68c94-dxqjf" Jul 28 22:45:09.356601 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:09.356521 2553 patch_prober.go:28] interesting pod/e2e-inf-a-104d62-openshift-default-7647657476-hbjqb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Jul 28 22:45:09.357016 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:09.356605 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-104d62-openshift-default-7647657476-hbjqb" podUID="89223ac8-2798-48fe-b8c8-5e255b7b0bf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Jul 28 22:45:09.465072 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:09.465041 2553 generic.go:356] "Generic (PLEG): container finished" podID="89223ac8-2798-48fe-b8c8-5e255b7b0bf2" containerID="afd79614dbb0638e66e48f03f53069b5f547839498b30a9aff53a66c6b1821b9" exitCode=0 Jul 28 22:45:09.465227 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:09.465119 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-104d62-openshift-default-7647657476-hbjqb" event={"ID":"89223ac8-2798-48fe-b8c8-5e255b7b0bf2","Type":"ContainerDied","Data":"afd79614dbb0638e66e48f03f53069b5f547839498b30a9aff53a66c6b1821b9"} Jul 28 22:45:09.465510 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:09.465497 2553 scope.go:117] "RemoveContainer" containerID="afd79614dbb0638e66e48f03f53069b5f547839498b30a9aff53a66c6b1821b9" Jul 28 22:45:09.466445 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:09.466423 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:45:09.466499 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:09.466487 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:45:09.466535 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:09.466519 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:45:10.472303 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:10.472267 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-104d62-openshift-default-7647657476-hbjqb" event={"ID":"89223ac8-2798-48fe-b8c8-5e255b7b0bf2","Type":"ContainerStarted","Data":"10bbdeb02d385794b087bff2ce12da8cd64fcb23a53e6c7c46111502cae38f0c"} Jul 28 22:45:10.554143 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:10.554099 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-104d62-openshift-default-7647657476-hbjqb" Jul 28 22:45:10.554411 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:10.554386 2553 patch_prober.go:28] interesting pod/e2e-inf-a-104d62-openshift-default-7647657476-hbjqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Jul 28 22:45:10.554458 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:10.554443 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-104d62-openshift-default-7647657476-hbjqb" podUID="89223ac8-2798-48fe-b8c8-5e255b7b0bf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Jul 28 22:45:11.553830 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:11.553788 2553 patch_prober.go:28] interesting pod/e2e-inf-a-104d62-openshift-default-7647657476-hbjqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Jul 28 22:45:11.554301 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:11.553848 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-104d62-openshift-default-7647657476-hbjqb" podUID="89223ac8-2798-48fe-b8c8-5e255b7b0bf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Jul 28 22:45:12.554612 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:12.554571 2553 patch_prober.go:28] interesting pod/e2e-inf-a-104d62-openshift-default-7647657476-hbjqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Jul 28 22:45:12.555046 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:12.554638 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-104d62-openshift-default-7647657476-hbjqb" podUID="89223ac8-2798-48fe-b8c8-5e255b7b0bf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Jul 28 22:45:13.554427 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:13.554394 2553 patch_prober.go:28] interesting pod/e2e-inf-a-104d62-openshift-default-7647657476-hbjqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Jul 28 22:45:13.554610 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:13.554445 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-104d62-openshift-default-7647657476-hbjqb" podUID="89223ac8-2798-48fe-b8c8-5e255b7b0bf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Jul 28 22:45:14.557949 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:14.557919 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-104d62-openshift-default-7647657476-hbjqb" Jul 28 22:45:14.558456 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:14.558302 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-104d62-openshift-default-7647657476-hbjqb" Jul 28 22:45:14.558970 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:14.558953 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-104d62-openshift-default-7647657476-hbjqb" Jul 28 22:45:21.067044 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:21.067012 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr"] Jul 28 22:45:21.069584 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:21.069565 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr" Jul 28 22:45:21.071800 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:21.071776 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-72fe47-openshift-default-dockercfg-wgfvp\"" Jul 28 22:45:21.081607 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:21.081580 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr"] Jul 28 22:45:21.220961 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:21.220932 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb-istio-token\") pod \"e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr\" (UID: \"049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb\") " pod="openshift-ingress/e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr" Jul 28 22:45:21.221146 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:21.220977 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb-istio-data\") pod \"e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr\" (UID: \"049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb\") " pod="openshift-ingress/e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr" Jul 28 22:45:21.221146 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:21.220997 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb-istio-podinfo\") pod \"e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr\" (UID: \"049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb\") " pod="openshift-ingress/e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr" Jul 28 22:45:21.221146 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:21.221097 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb-istio-envoy\") pod \"e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr\" (UID: \"049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb\") " pod="openshift-ingress/e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr" Jul 28 22:45:21.221146 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:21.221138 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb-workload-socket\") pod \"e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr\" (UID: \"049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb\") " pod="openshift-ingress/e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr" Jul 28 22:45:21.221381 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:21.221178 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mlw8\" (UniqueName: \"kubernetes.io/projected/049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb-kube-api-access-2mlw8\") pod \"e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr\" (UID: \"049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb\") " pod="openshift-ingress/e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr" Jul 28 22:45:21.221381 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:21.221258 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb-istiod-ca-cert\") pod \"e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr\" (UID: \"049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb\") " pod="openshift-ingress/e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr" Jul 28 22:45:21.221381 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:21.221312 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb-credential-socket\") pod \"e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr\" (UID: \"049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb\") " pod="openshift-ingress/e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr" Jul 28 22:45:21.221381 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:21.221356 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb-workload-certs\") pod \"e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr\" (UID: \"049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb\") " pod="openshift-ingress/e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr" Jul 28 22:45:21.322311 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:21.322229 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb-workload-socket\") pod \"e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr\" (UID: \"049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb\") " pod="openshift-ingress/e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr" Jul 28 22:45:21.322311 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:21.322277 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mlw8\" (UniqueName: \"kubernetes.io/projected/049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb-kube-api-access-2mlw8\") pod \"e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr\" (UID: \"049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb\") " pod="openshift-ingress/e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr" Jul 28 22:45:21.322311 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:21.322301 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb-istiod-ca-cert\") pod \"e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr\" (UID: \"049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb\") " pod="openshift-ingress/e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr" Jul 28 22:45:21.322679 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:21.322326 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb-credential-socket\") pod \"e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr\" (UID: \"049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb\") " pod="openshift-ingress/e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr" Jul 28 22:45:21.322679 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:21.322352 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb-workload-certs\") pod \"e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr\" (UID: \"049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb\") " pod="openshift-ingress/e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr" Jul 28 22:45:21.322679 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:21.322370 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb-istio-token\") pod \"e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr\" (UID: \"049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb\") " pod="openshift-ingress/e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr" Jul 28 22:45:21.322679 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:21.322413 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb-istio-data\") pod \"e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr\" (UID: \"049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb\") " pod="openshift-ingress/e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr" Jul 28 22:45:21.322679 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:21.322434 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb-istio-podinfo\") pod \"e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr\" (UID: \"049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb\") " pod="openshift-ingress/e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr" Jul 28 22:45:21.322679 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:21.322471 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb-istio-envoy\") pod \"e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr\" (UID: \"049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb\") " pod="openshift-ingress/e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr" Jul 28 22:45:21.322679 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:21.322632 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb-workload-socket\") pod \"e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr\" (UID: \"049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb\") " pod="openshift-ingress/e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr" Jul 28 22:45:21.323145 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:21.323122 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb-credential-socket\") pod \"e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr\" (UID: \"049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb\") " pod="openshift-ingress/e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr" Jul 28 22:45:21.323232 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:21.323122 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb-istiod-ca-cert\") pod \"e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr\" (UID: \"049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb\") " pod="openshift-ingress/e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr" Jul 28 22:45:21.323232 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:21.323212 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb-workload-certs\") pod \"e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr\" (UID: \"049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb\") " pod="openshift-ingress/e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr" Jul 28 22:45:21.323329 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:21.323192 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb-istio-data\") pod \"e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr\" (UID: \"049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb\") " pod="openshift-ingress/e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr" Jul 28 22:45:21.325065 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:21.325046 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb-istio-envoy\") pod \"e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr\" (UID: \"049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb\") " pod="openshift-ingress/e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr" Jul 28 22:45:21.325174 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:21.325070 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb-istio-podinfo\") pod \"e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr\" (UID: \"049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb\") " pod="openshift-ingress/e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr" Jul 28 22:45:21.330185 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:21.330165 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mlw8\" (UniqueName: \"kubernetes.io/projected/049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb-kube-api-access-2mlw8\") pod \"e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr\" (UID: \"049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb\") " pod="openshift-ingress/e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr" Jul 28 22:45:21.330297 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:21.330255 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb-istio-token\") pod \"e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr\" (UID: \"049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb\") " pod="openshift-ingress/e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr" Jul 28 22:45:21.380438 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:21.380406 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr" Jul 28 22:45:21.519010 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:21.518981 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr"] Jul 28 22:45:21.520737 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:45:21.520683 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod049c2f5a_16ba_48b4_b6f3_6f3ae4bcbedb.slice/crio-3dc36ed2181b9ffa09c8b08342979fde7e263e8997d3e4346d6ba4b778ed3a06 WatchSource:0}: Error finding container 3dc36ed2181b9ffa09c8b08342979fde7e263e8997d3e4346d6ba4b778ed3a06: Status 404 returned error can't find the container with id 3dc36ed2181b9ffa09c8b08342979fde7e263e8997d3e4346d6ba4b778ed3a06 Jul 28 22:45:21.523036 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:21.523000 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:45:21.523113 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:21.523091 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:45:21.523165 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:21.523137 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:45:21.530408 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:21.530385 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr" event={"ID":"049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb","Type":"ContainerStarted","Data":"3dc36ed2181b9ffa09c8b08342979fde7e263e8997d3e4346d6ba4b778ed3a06"} Jul 28 22:45:22.536911 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:22.536873 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr" event={"ID":"049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb","Type":"ContainerStarted","Data":"d408f39f38502858d706553286e63b620504c89409a460b305fc5b89867e973a"} Jul 28 22:45:22.555486 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:22.555432 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr" podStartSLOduration=1.555415711 podStartE2EDuration="1.555415711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:45:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:45:22.554138007 +0000 UTC m=+2142.641396092" watchObservedRunningTime="2026-07-28 22:45:22.555415711 +0000 UTC m=+2142.642673779" Jul 28 22:45:23.381066 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:23.381025 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr" Jul 28 22:45:23.386045 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:23.386019 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr" Jul 28 22:45:23.543319 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:23.543273 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr" Jul 28 22:45:23.544398 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:23.544376 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr" Jul 28 22:45:24.674883 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:24.674846 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-72fe47-8bfb7984f-6bhd9"] Jul 28 22:45:24.677923 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:24.677897 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72fe47-8bfb7984f-6bhd9" Jul 28 22:45:24.680546 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:24.680522 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-72fe47-serving-cert\"" Jul 28 22:45:24.694031 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:24.694008 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-72fe47-8bfb7984f-6bhd9"] Jul 28 22:45:24.855633 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:24.855595 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/242754b4-5708-477d-8069-01f034970ad7-maas-api-tls\") pod \"maas-api-e2e-inf-b-72fe47-8bfb7984f-6bhd9\" (UID: \"242754b4-5708-477d-8069-01f034970ad7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72fe47-8bfb7984f-6bhd9" Jul 28 22:45:24.855864 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:24.855681 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/242754b4-5708-477d-8069-01f034970ad7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-72fe47-8bfb7984f-6bhd9\" (UID: \"242754b4-5708-477d-8069-01f034970ad7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72fe47-8bfb7984f-6bhd9" Jul 28 22:45:24.855864 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:24.855788 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks79v\" (UniqueName: \"kubernetes.io/projected/242754b4-5708-477d-8069-01f034970ad7-kube-api-access-ks79v\") pod \"maas-api-e2e-inf-b-72fe47-8bfb7984f-6bhd9\" (UID: \"242754b4-5708-477d-8069-01f034970ad7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72fe47-8bfb7984f-6bhd9" Jul 28 22:45:24.957036 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:24.956939 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks79v\" (UniqueName: \"kubernetes.io/projected/242754b4-5708-477d-8069-01f034970ad7-kube-api-access-ks79v\") pod \"maas-api-e2e-inf-b-72fe47-8bfb7984f-6bhd9\" (UID: \"242754b4-5708-477d-8069-01f034970ad7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72fe47-8bfb7984f-6bhd9" Jul 28 22:45:24.957036 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:24.956989 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/242754b4-5708-477d-8069-01f034970ad7-maas-api-tls\") pod \"maas-api-e2e-inf-b-72fe47-8bfb7984f-6bhd9\" (UID: \"242754b4-5708-477d-8069-01f034970ad7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72fe47-8bfb7984f-6bhd9" Jul 28 22:45:24.957269 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:24.957062 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/242754b4-5708-477d-8069-01f034970ad7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-72fe47-8bfb7984f-6bhd9\" (UID: \"242754b4-5708-477d-8069-01f034970ad7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72fe47-8bfb7984f-6bhd9" Jul 28 22:45:24.957802 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:24.957777 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/242754b4-5708-477d-8069-01f034970ad7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-72fe47-8bfb7984f-6bhd9\" (UID: \"242754b4-5708-477d-8069-01f034970ad7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72fe47-8bfb7984f-6bhd9" Jul 28 22:45:24.959640 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:24.959621 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/242754b4-5708-477d-8069-01f034970ad7-maas-api-tls\") pod \"maas-api-e2e-inf-b-72fe47-8bfb7984f-6bhd9\" (UID: \"242754b4-5708-477d-8069-01f034970ad7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72fe47-8bfb7984f-6bhd9" Jul 28 22:45:24.972896 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:24.972870 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks79v\" (UniqueName: \"kubernetes.io/projected/242754b4-5708-477d-8069-01f034970ad7-kube-api-access-ks79v\") pod \"maas-api-e2e-inf-b-72fe47-8bfb7984f-6bhd9\" (UID: \"242754b4-5708-477d-8069-01f034970ad7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72fe47-8bfb7984f-6bhd9" Jul 28 22:45:24.988741 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:24.988714 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72fe47-8bfb7984f-6bhd9" Jul 28 22:45:25.128878 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:25.128841 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-72fe47-8bfb7984f-6bhd9"] Jul 28 22:45:25.129864 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:45:25.129839 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod242754b4_5708_477d_8069_01f034970ad7.slice/crio-04acf731823cbec71ec37bbcd75573ce7334e4c2dcc03696928c81f798c5118e WatchSource:0}: Error finding container 04acf731823cbec71ec37bbcd75573ce7334e4c2dcc03696928c81f798c5118e: Status 404 returned error can't find the container with id 04acf731823cbec71ec37bbcd75573ce7334e4c2dcc03696928c81f798c5118e Jul 28 22:45:25.555620 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:25.555590 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72fe47-8bfb7984f-6bhd9" event={"ID":"242754b4-5708-477d-8069-01f034970ad7","Type":"ContainerStarted","Data":"5ee1830e588353e2ee02028c7dff17be4a6be01fb8fbcdc5ecfc7bfe17cd8614"} Jul 28 22:45:25.555768 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:25.555652 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72fe47-8bfb7984f-6bhd9" event={"ID":"242754b4-5708-477d-8069-01f034970ad7","Type":"ContainerStarted","Data":"04acf731823cbec71ec37bbcd75573ce7334e4c2dcc03696928c81f798c5118e"} Jul 28 22:45:25.555838 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:25.555825 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72fe47-8bfb7984f-6bhd9" Jul 28 22:45:25.585296 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:25.585230 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72fe47-8bfb7984f-6bhd9" podStartSLOduration=1.231561847 podStartE2EDuration="1.585208934s" totalImagesPullingTime="353.647087ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:45:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:45:25.58131107 +0000 UTC m=+2145.668569142" watchObservedRunningTime="2026-07-28 22:45:25.585208934 +0000 UTC m=+2145.672467004" Jul 28 22:45:31.566973 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:31.566892 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72fe47-8bfb7984f-6bhd9" Jul 28 22:45:39.584860 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:39.584826 2553 patch_prober.go:28] interesting pod/e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 28 22:45:39.585324 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:39.584888 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr" podUID="049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 28 22:45:40.642456 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:40.642369 2553 generic.go:356] "Generic (PLEG): container finished" podID="049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb" containerID="d408f39f38502858d706553286e63b620504c89409a460b305fc5b89867e973a" exitCode=0 Jul 28 22:45:40.642456 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:40.642440 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr" event={"ID":"049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb","Type":"ContainerDied","Data":"d408f39f38502858d706553286e63b620504c89409a460b305fc5b89867e973a"} Jul 28 22:45:40.642983 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:40.642858 2553 scope.go:117] "RemoveContainer" containerID="d408f39f38502858d706553286e63b620504c89409a460b305fc5b89867e973a" Jul 28 22:45:40.643879 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:40.643851 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:45:40.643969 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:40.643915 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:45:40.643969 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:40.643946 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:45:41.653894 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:41.653842 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr" event={"ID":"049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb","Type":"ContainerStarted","Data":"ca24a2484164b11f86257c816f89edb32c8e97066b937573ffc5f69a7a9aeca9"} Jul 28 22:45:42.381051 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:42.381003 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr" Jul 28 22:45:42.381247 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:42.381217 2553 patch_prober.go:28] interesting pod/e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 28 22:45:42.381295 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:42.381281 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr" podUID="049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 28 22:45:43.381323 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:43.381281 2553 patch_prober.go:28] interesting pod/e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 28 22:45:43.381769 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:43.381374 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr" podUID="049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 28 22:45:44.381250 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:44.381211 2553 patch_prober.go:28] interesting pod/e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 28 22:45:44.381431 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:44.381271 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr" podUID="049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 28 22:45:45.381694 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:45.381656 2553 patch_prober.go:28] interesting pod/e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 28 22:45:45.382100 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:45.381737 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr" podUID="049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 28 22:45:46.384638 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:46.384608 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr" Jul 28 22:45:46.385129 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:46.385032 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr" Jul 28 22:45:46.385677 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:45:46.385656 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr" Jul 28 22:47:00.170086 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:00.170042 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-104d62-m489l"] Jul 28 22:47:00.173240 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:00.173218 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-104d62-m489l" Jul 28 22:47:00.175301 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:00.175275 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qf46x\"" Jul 28 22:47:00.179439 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:00.179415 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-104d62-m489l"] Jul 28 22:47:00.353551 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:00.353509 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4512921a-6fa4-4475-bc64-85169d264b10-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-104d62-m489l\" (UID: \"4512921a-6fa4-4475-bc64-85169d264b10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-104d62-m489l" Jul 28 22:47:00.455173 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:00.455084 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4512921a-6fa4-4475-bc64-85169d264b10-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-104d62-m489l\" (UID: \"4512921a-6fa4-4475-bc64-85169d264b10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-104d62-m489l" Jul 28 22:47:00.455797 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:00.455776 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4512921a-6fa4-4475-bc64-85169d264b10-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-104d62-m489l\" (UID: \"4512921a-6fa4-4475-bc64-85169d264b10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-104d62-m489l" Jul 28 22:47:00.486037 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:00.486016 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-104d62-m489l" Jul 28 22:47:00.619544 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:00.619516 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-104d62-m489l"] Jul 28 22:47:00.621613 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:47:00.621590 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4512921a_6fa4_4475_bc64_85169d264b10.slice/crio-ae75e9e80d6c74af31257c8cfcf61849838ab0f70c7503e41e260d31446d00ff WatchSource:0}: Error finding container ae75e9e80d6c74af31257c8cfcf61849838ab0f70c7503e41e260d31446d00ff: Status 404 returned error can't find the container with id ae75e9e80d6c74af31257c8cfcf61849838ab0f70c7503e41e260d31446d00ff Jul 28 22:47:01.094073 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:01.093974 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-104d62-m489l" event={"ID":"4512921a-6fa4-4475-bc64-85169d264b10","Type":"ContainerStarted","Data":"01fe2f4e036daf2b7fc7b20572c6f7b6d179b2f3717e3626bfe34ac2895f7a73"} Jul 28 22:47:01.094073 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:01.094026 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-104d62-m489l" event={"ID":"4512921a-6fa4-4475-bc64-85169d264b10","Type":"ContainerStarted","Data":"ae75e9e80d6c74af31257c8cfcf61849838ab0f70c7503e41e260d31446d00ff"} Jul 28 22:47:01.109763 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:01.109616 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-104d62-m489l" podStartSLOduration=1.109585817 podStartE2EDuration="1.109585817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:47:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:47:01.106270739 +0000 UTC m=+2241.193528811" watchObservedRunningTime="2026-07-28 22:47:01.109585817 +0000 UTC m=+2241.196843888" Jul 28 22:47:02.101829 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:02.101789 2553 generic.go:356] "Generic (PLEG): container finished" podID="4512921a-6fa4-4475-bc64-85169d264b10" containerID="01fe2f4e036daf2b7fc7b20572c6f7b6d179b2f3717e3626bfe34ac2895f7a73" exitCode=0 Jul 28 22:47:02.102239 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:02.101877 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-104d62-m489l" event={"ID":"4512921a-6fa4-4475-bc64-85169d264b10","Type":"ContainerDied","Data":"01fe2f4e036daf2b7fc7b20572c6f7b6d179b2f3717e3626bfe34ac2895f7a73"} Jul 28 22:47:03.253467 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:03.253442 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-104d62-m489l" Jul 28 22:47:03.282596 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:03.282564 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4512921a-6fa4-4475-bc64-85169d264b10-maas-api-service-ca\") pod \"4512921a-6fa4-4475-bc64-85169d264b10\" (UID: \"4512921a-6fa4-4475-bc64-85169d264b10\") " Jul 28 22:47:03.282966 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:03.282942 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4512921a-6fa4-4475-bc64-85169d264b10-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4512921a-6fa4-4475-bc64-85169d264b10" (UID: "4512921a-6fa4-4475-bc64-85169d264b10"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 22:47:03.383827 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:03.383736 2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4512921a-6fa4-4475-bc64-85169d264b10-maas-api-service-ca\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:47:04.114654 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:04.114624 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-104d62-m489l" Jul 28 22:47:04.114654 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:04.114638 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-104d62-m489l" event={"ID":"4512921a-6fa4-4475-bc64-85169d264b10","Type":"ContainerDied","Data":"ae75e9e80d6c74af31257c8cfcf61849838ab0f70c7503e41e260d31446d00ff"} Jul 28 22:47:04.114884 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:04.114684 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae75e9e80d6c74af31257c8cfcf61849838ab0f70c7503e41e260d31446d00ff" Jul 28 22:47:10.633359 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:10.633323 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-104d62-5df9b68c94-dxqjf"] Jul 28 22:47:10.633853 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:10.633588 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-104d62-5df9b68c94-dxqjf" podUID="83e80fde-c25b-410e-9523-281a4a0450c5" containerName="maas-api" containerID="cri-o://7bef1945abb3bdcd1a15fa4fba07a410f64a52f0da4fc2f630dae93484900104" gracePeriod=30 Jul 28 22:47:10.944303 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:10.944275 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-104d62-m489l"] Jul 28 22:47:10.948877 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:10.948842 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-104d62-m489l"] Jul 28 22:47:11.099119 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:11.099090 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-104d62-5df9b68c94-dxqjf" Jul 28 22:47:11.151142 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:11.151114 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83e80fde-c25b-410e-9523-281a4a0450c5-openshift-service-ca\") pod \"83e80fde-c25b-410e-9523-281a4a0450c5\" (UID: \"83e80fde-c25b-410e-9523-281a4a0450c5\") " Jul 28 22:47:11.151327 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:11.151227 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9pklx\" (UniqueName: \"kubernetes.io/projected/83e80fde-c25b-410e-9523-281a4a0450c5-kube-api-access-9pklx\") pod \"83e80fde-c25b-410e-9523-281a4a0450c5\" (UID: \"83e80fde-c25b-410e-9523-281a4a0450c5\") " Jul 28 22:47:11.151327 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:11.151282 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83e80fde-c25b-410e-9523-281a4a0450c5-maas-api-tls\") pod \"83e80fde-c25b-410e-9523-281a4a0450c5\" (UID: \"83e80fde-c25b-410e-9523-281a4a0450c5\") " Jul 28 22:47:11.151521 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:11.151498 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83e80fde-c25b-410e-9523-281a4a0450c5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "83e80fde-c25b-410e-9523-281a4a0450c5" (UID: "83e80fde-c25b-410e-9523-281a4a0450c5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 22:47:11.151627 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:11.151606 2553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83e80fde-c25b-410e-9523-281a4a0450c5-openshift-service-ca\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:47:11.155456 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:11.154053 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83e80fde-c25b-410e-9523-281a4a0450c5-kube-api-access-9pklx" (OuterVolumeSpecName: "kube-api-access-9pklx") pod "83e80fde-c25b-410e-9523-281a4a0450c5" (UID: "83e80fde-c25b-410e-9523-281a4a0450c5"). InnerVolumeSpecName "kube-api-access-9pklx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:47:11.155456 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:11.154097 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83e80fde-c25b-410e-9523-281a4a0450c5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "83e80fde-c25b-410e-9523-281a4a0450c5" (UID: "83e80fde-c25b-410e-9523-281a4a0450c5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 22:47:11.155456 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:11.154215 2553 generic.go:356] "Generic (PLEG): container finished" podID="83e80fde-c25b-410e-9523-281a4a0450c5" containerID="7bef1945abb3bdcd1a15fa4fba07a410f64a52f0da4fc2f630dae93484900104" exitCode=0 Jul 28 22:47:11.155456 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:11.154320 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-104d62-5df9b68c94-dxqjf" event={"ID":"83e80fde-c25b-410e-9523-281a4a0450c5","Type":"ContainerDied","Data":"7bef1945abb3bdcd1a15fa4fba07a410f64a52f0da4fc2f630dae93484900104"} Jul 28 22:47:11.155456 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:11.154401 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-104d62-5df9b68c94-dxqjf" event={"ID":"83e80fde-c25b-410e-9523-281a4a0450c5","Type":"ContainerDied","Data":"db36cc7819f1dfdc931bce02d804c8f758b189e3f79ab5d382437a00b8085ea0"} Jul 28 22:47:11.155456 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:11.154447 2553 scope.go:117] "RemoveContainer" containerID="7bef1945abb3bdcd1a15fa4fba07a410f64a52f0da4fc2f630dae93484900104" Jul 28 22:47:11.155456 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:11.154460 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-104d62-5df9b68c94-dxqjf" Jul 28 22:47:11.178996 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:11.178972 2553 scope.go:117] "RemoveContainer" containerID="7bef1945abb3bdcd1a15fa4fba07a410f64a52f0da4fc2f630dae93484900104" Jul 28 22:47:11.179327 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:47:11.179305 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bef1945abb3bdcd1a15fa4fba07a410f64a52f0da4fc2f630dae93484900104\": container with ID starting with 7bef1945abb3bdcd1a15fa4fba07a410f64a52f0da4fc2f630dae93484900104 not found: ID does not exist" containerID="7bef1945abb3bdcd1a15fa4fba07a410f64a52f0da4fc2f630dae93484900104" Jul 28 22:47:11.179377 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:11.179336 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bef1945abb3bdcd1a15fa4fba07a410f64a52f0da4fc2f630dae93484900104"} err="failed to get container status \"7bef1945abb3bdcd1a15fa4fba07a410f64a52f0da4fc2f630dae93484900104\": rpc error: code = NotFound desc = could not find container \"7bef1945abb3bdcd1a15fa4fba07a410f64a52f0da4fc2f630dae93484900104\": container with ID starting with 7bef1945abb3bdcd1a15fa4fba07a410f64a52f0da4fc2f630dae93484900104 not found: ID does not exist" Jul 28 22:47:11.186357 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:11.186335 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-104d62-5df9b68c94-dxqjf"] Jul 28 22:47:11.192486 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:11.192464 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-104d62-5df9b68c94-dxqjf"] Jul 28 22:47:11.252871 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:11.252836 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pklx\" (UniqueName: \"kubernetes.io/projected/83e80fde-c25b-410e-9523-281a4a0450c5-kube-api-access-9pklx\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:47:11.252871 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:11.252864 2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83e80fde-c25b-410e-9523-281a4a0450c5-maas-api-tls\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:47:12.405662 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:12.405625 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4512921a-6fa4-4475-bc64-85169d264b10" path="/var/lib/kubelet/pods/4512921a-6fa4-4475-bc64-85169d264b10/volumes" Jul 28 22:47:12.406072 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:12.406052 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83e80fde-c25b-410e-9523-281a4a0450c5" path="/var/lib/kubelet/pods/83e80fde-c25b-410e-9523-281a4a0450c5/volumes" Jul 28 22:47:17.358649 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:17.358617 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-104d62-openshift-default-7647657476-hbjqb"] Jul 28 22:47:17.359052 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:17.358907 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-104d62-openshift-default-7647657476-hbjqb" podUID="89223ac8-2798-48fe-b8c8-5e255b7b0bf2" containerName="istio-proxy" containerID="cri-o://10bbdeb02d385794b087bff2ce12da8cd64fcb23a53e6c7c46111502cae38f0c" gracePeriod=30 Jul 28 22:47:22.622468 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:22.622396 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-104d62-openshift-default-7647657476-hbjqb" Jul 28 22:47:22.754556 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:22.754524 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89223ac8-2798-48fe-b8c8-5e255b7b0bf2-istio-envoy\") pod \"89223ac8-2798-48fe-b8c8-5e255b7b0bf2\" (UID: \"89223ac8-2798-48fe-b8c8-5e255b7b0bf2\") " Jul 28 22:47:22.754556 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:22.754562 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89223ac8-2798-48fe-b8c8-5e255b7b0bf2-istio-podinfo\") pod \"89223ac8-2798-48fe-b8c8-5e255b7b0bf2\" (UID: \"89223ac8-2798-48fe-b8c8-5e255b7b0bf2\") " Jul 28 22:47:22.754841 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:22.754608 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89223ac8-2798-48fe-b8c8-5e255b7b0bf2-istiod-ca-cert\") pod \"89223ac8-2798-48fe-b8c8-5e255b7b0bf2\" (UID: \"89223ac8-2798-48fe-b8c8-5e255b7b0bf2\") " Jul 28 22:47:22.754841 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:22.754638 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89223ac8-2798-48fe-b8c8-5e255b7b0bf2-workload-socket\") pod \"89223ac8-2798-48fe-b8c8-5e255b7b0bf2\" (UID: \"89223ac8-2798-48fe-b8c8-5e255b7b0bf2\") " Jul 28 22:47:22.754841 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:22.754727 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89223ac8-2798-48fe-b8c8-5e255b7b0bf2-workload-certs\") pod \"89223ac8-2798-48fe-b8c8-5e255b7b0bf2\" (UID: \"89223ac8-2798-48fe-b8c8-5e255b7b0bf2\") " Jul 28 22:47:22.754841 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:22.754790 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9gwhq\" (UniqueName: \"kubernetes.io/projected/89223ac8-2798-48fe-b8c8-5e255b7b0bf2-kube-api-access-9gwhq\") pod \"89223ac8-2798-48fe-b8c8-5e255b7b0bf2\" (UID: \"89223ac8-2798-48fe-b8c8-5e255b7b0bf2\") " Jul 28 22:47:22.754841 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:22.754816 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89223ac8-2798-48fe-b8c8-5e255b7b0bf2-credential-socket\") pod \"89223ac8-2798-48fe-b8c8-5e255b7b0bf2\" (UID: \"89223ac8-2798-48fe-b8c8-5e255b7b0bf2\") " Jul 28 22:47:22.754841 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:22.754843 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89223ac8-2798-48fe-b8c8-5e255b7b0bf2-istio-token\") pod \"89223ac8-2798-48fe-b8c8-5e255b7b0bf2\" (UID: \"89223ac8-2798-48fe-b8c8-5e255b7b0bf2\") " Jul 28 22:47:22.755150 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:22.754882 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89223ac8-2798-48fe-b8c8-5e255b7b0bf2-istio-data\") pod \"89223ac8-2798-48fe-b8c8-5e255b7b0bf2\" (UID: \"89223ac8-2798-48fe-b8c8-5e255b7b0bf2\") " Jul 28 22:47:22.755150 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:22.755018 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89223ac8-2798-48fe-b8c8-5e255b7b0bf2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "89223ac8-2798-48fe-b8c8-5e255b7b0bf2" (UID: "89223ac8-2798-48fe-b8c8-5e255b7b0bf2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:47:22.755150 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:22.755111 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89223ac8-2798-48fe-b8c8-5e255b7b0bf2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "89223ac8-2798-48fe-b8c8-5e255b7b0bf2" (UID: "89223ac8-2798-48fe-b8c8-5e255b7b0bf2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:47:22.755150 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:22.755110 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89223ac8-2798-48fe-b8c8-5e255b7b0bf2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "89223ac8-2798-48fe-b8c8-5e255b7b0bf2" (UID: "89223ac8-2798-48fe-b8c8-5e255b7b0bf2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 22:47:22.755406 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:22.755353 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89223ac8-2798-48fe-b8c8-5e255b7b0bf2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "89223ac8-2798-48fe-b8c8-5e255b7b0bf2" (UID: "89223ac8-2798-48fe-b8c8-5e255b7b0bf2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:47:22.755406 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:22.755372 2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89223ac8-2798-48fe-b8c8-5e255b7b0bf2-istiod-ca-cert\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:47:22.755406 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:22.755393 2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89223ac8-2798-48fe-b8c8-5e255b7b0bf2-workload-socket\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:47:22.755406 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:22.755409 2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89223ac8-2798-48fe-b8c8-5e255b7b0bf2-workload-certs\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:47:22.756185 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:22.756158 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89223ac8-2798-48fe-b8c8-5e255b7b0bf2-istio-data" (OuterVolumeSpecName: "istio-data") pod "89223ac8-2798-48fe-b8c8-5e255b7b0bf2" (UID: "89223ac8-2798-48fe-b8c8-5e255b7b0bf2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:47:22.757088 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:22.757064 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89223ac8-2798-48fe-b8c8-5e255b7b0bf2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "89223ac8-2798-48fe-b8c8-5e255b7b0bf2" (UID: "89223ac8-2798-48fe-b8c8-5e255b7b0bf2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:47:22.757088 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:22.757075 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/89223ac8-2798-48fe-b8c8-5e255b7b0bf2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "89223ac8-2798-48fe-b8c8-5e255b7b0bf2" (UID: "89223ac8-2798-48fe-b8c8-5e255b7b0bf2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 22:47:22.757216 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:22.757187 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89223ac8-2798-48fe-b8c8-5e255b7b0bf2-kube-api-access-9gwhq" (OuterVolumeSpecName: "kube-api-access-9gwhq") pod "89223ac8-2798-48fe-b8c8-5e255b7b0bf2" (UID: "89223ac8-2798-48fe-b8c8-5e255b7b0bf2"). InnerVolumeSpecName "kube-api-access-9gwhq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:47:22.757276 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:22.757262 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89223ac8-2798-48fe-b8c8-5e255b7b0bf2-istio-token" (OuterVolumeSpecName: "istio-token") pod "89223ac8-2798-48fe-b8c8-5e255b7b0bf2" (UID: "89223ac8-2798-48fe-b8c8-5e255b7b0bf2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:47:22.855854 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:22.855817 2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89223ac8-2798-48fe-b8c8-5e255b7b0bf2-credential-socket\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:47:22.855854 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:22.855851 2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89223ac8-2798-48fe-b8c8-5e255b7b0bf2-istio-token\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:47:22.855854 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:22.855860 2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89223ac8-2798-48fe-b8c8-5e255b7b0bf2-istio-data\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:47:22.856084 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:22.855867 2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89223ac8-2798-48fe-b8c8-5e255b7b0bf2-istio-envoy\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:47:22.856084 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:22.855875 2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89223ac8-2798-48fe-b8c8-5e255b7b0bf2-istio-podinfo\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:47:22.856084 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:22.855883 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gwhq\" (UniqueName: \"kubernetes.io/projected/89223ac8-2798-48fe-b8c8-5e255b7b0bf2-kube-api-access-9gwhq\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:47:23.228687 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:23.228649 2553 generic.go:356] "Generic (PLEG): container finished" podID="89223ac8-2798-48fe-b8c8-5e255b7b0bf2" containerID="10bbdeb02d385794b087bff2ce12da8cd64fcb23a53e6c7c46111502cae38f0c" exitCode=0 Jul 28 22:47:23.228882 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:23.228732 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-104d62-openshift-default-7647657476-hbjqb" event={"ID":"89223ac8-2798-48fe-b8c8-5e255b7b0bf2","Type":"ContainerDied","Data":"10bbdeb02d385794b087bff2ce12da8cd64fcb23a53e6c7c46111502cae38f0c"} Jul 28 22:47:23.228882 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:23.228810 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-104d62-openshift-default-7647657476-hbjqb" event={"ID":"89223ac8-2798-48fe-b8c8-5e255b7b0bf2","Type":"ContainerDied","Data":"4656bc2ba7504983feb5a3ed8d76c4377ec73cf7da9fabd2757ee5c57b62f3d0"} Jul 28 22:47:23.228882 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:23.228739 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-104d62-openshift-default-7647657476-hbjqb" Jul 28 22:47:23.228882 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:23.228832 2553 scope.go:117] "RemoveContainer" containerID="10bbdeb02d385794b087bff2ce12da8cd64fcb23a53e6c7c46111502cae38f0c" Jul 28 22:47:23.239143 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:23.239127 2553 scope.go:117] "RemoveContainer" containerID="afd79614dbb0638e66e48f03f53069b5f547839498b30a9aff53a66c6b1821b9" Jul 28 22:47:23.247878 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:23.247854 2553 scope.go:117] "RemoveContainer" containerID="10bbdeb02d385794b087bff2ce12da8cd64fcb23a53e6c7c46111502cae38f0c" Jul 28 22:47:23.248193 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:47:23.248171 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10bbdeb02d385794b087bff2ce12da8cd64fcb23a53e6c7c46111502cae38f0c\": container with ID starting with 10bbdeb02d385794b087bff2ce12da8cd64fcb23a53e6c7c46111502cae38f0c not found: ID does not exist" containerID="10bbdeb02d385794b087bff2ce12da8cd64fcb23a53e6c7c46111502cae38f0c" Jul 28 22:47:23.248288 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:23.248203 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10bbdeb02d385794b087bff2ce12da8cd64fcb23a53e6c7c46111502cae38f0c"} err="failed to get container status \"10bbdeb02d385794b087bff2ce12da8cd64fcb23a53e6c7c46111502cae38f0c\": rpc error: code = NotFound desc = could not find container \"10bbdeb02d385794b087bff2ce12da8cd64fcb23a53e6c7c46111502cae38f0c\": container with ID starting with 10bbdeb02d385794b087bff2ce12da8cd64fcb23a53e6c7c46111502cae38f0c not found: ID does not exist" Jul 28 22:47:23.248288 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:23.248227 2553 scope.go:117] "RemoveContainer" containerID="afd79614dbb0638e66e48f03f53069b5f547839498b30a9aff53a66c6b1821b9" Jul 28 22:47:23.248490 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:47:23.248473 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afd79614dbb0638e66e48f03f53069b5f547839498b30a9aff53a66c6b1821b9\": container with ID starting with afd79614dbb0638e66e48f03f53069b5f547839498b30a9aff53a66c6b1821b9 not found: ID does not exist" containerID="afd79614dbb0638e66e48f03f53069b5f547839498b30a9aff53a66c6b1821b9" Jul 28 22:47:23.248535 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:23.248496 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afd79614dbb0638e66e48f03f53069b5f547839498b30a9aff53a66c6b1821b9"} err="failed to get container status \"afd79614dbb0638e66e48f03f53069b5f547839498b30a9aff53a66c6b1821b9\": rpc error: code = NotFound desc = could not find container \"afd79614dbb0638e66e48f03f53069b5f547839498b30a9aff53a66c6b1821b9\": container with ID starting with afd79614dbb0638e66e48f03f53069b5f547839498b30a9aff53a66c6b1821b9 not found: ID does not exist" Jul 28 22:47:23.249961 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:23.249941 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-104d62-openshift-default-7647657476-hbjqb"] Jul 28 22:47:23.255729 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:23.255688 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-104d62-openshift-default-7647657476-hbjqb"] Jul 28 22:47:24.405384 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:24.405350 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89223ac8-2798-48fe-b8c8-5e255b7b0bf2" path="/var/lib/kubelet/pods/89223ac8-2798-48fe-b8c8-5e255b7b0bf2/volumes" Jul 28 22:47:28.102849 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:28.102817 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-72fe47-8bfb7984f-6bhd9"] Jul 28 22:47:28.103376 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:28.103035 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72fe47-8bfb7984f-6bhd9" podUID="242754b4-5708-477d-8069-01f034970ad7" containerName="maas-api" containerID="cri-o://5ee1830e588353e2ee02028c7dff17be4a6be01fb8fbcdc5ecfc7bfe17cd8614" gracePeriod=30 Jul 28 22:47:28.259007 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:28.258972 2553 generic.go:356] "Generic (PLEG): container finished" podID="242754b4-5708-477d-8069-01f034970ad7" containerID="5ee1830e588353e2ee02028c7dff17be4a6be01fb8fbcdc5ecfc7bfe17cd8614" exitCode=0 Jul 28 22:47:28.259172 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:28.259040 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72fe47-8bfb7984f-6bhd9" event={"ID":"242754b4-5708-477d-8069-01f034970ad7","Type":"ContainerDied","Data":"5ee1830e588353e2ee02028c7dff17be4a6be01fb8fbcdc5ecfc7bfe17cd8614"} Jul 28 22:47:28.359070 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:28.359001 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72fe47-8bfb7984f-6bhd9" Jul 28 22:47:28.506463 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:28.506428 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/242754b4-5708-477d-8069-01f034970ad7-maas-api-tls\") pod \"242754b4-5708-477d-8069-01f034970ad7\" (UID: \"242754b4-5708-477d-8069-01f034970ad7\") " Jul 28 22:47:28.506622 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:28.506524 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ks79v\" (UniqueName: \"kubernetes.io/projected/242754b4-5708-477d-8069-01f034970ad7-kube-api-access-ks79v\") pod \"242754b4-5708-477d-8069-01f034970ad7\" (UID: \"242754b4-5708-477d-8069-01f034970ad7\") " Jul 28 22:47:28.506622 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:28.506555 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/242754b4-5708-477d-8069-01f034970ad7-openshift-service-ca\") pod \"242754b4-5708-477d-8069-01f034970ad7\" (UID: \"242754b4-5708-477d-8069-01f034970ad7\") " Jul 28 22:47:28.506983 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:28.506958 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/242754b4-5708-477d-8069-01f034970ad7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "242754b4-5708-477d-8069-01f034970ad7" (UID: "242754b4-5708-477d-8069-01f034970ad7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 22:47:28.508857 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:28.508833 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/242754b4-5708-477d-8069-01f034970ad7-kube-api-access-ks79v" (OuterVolumeSpecName: "kube-api-access-ks79v") pod "242754b4-5708-477d-8069-01f034970ad7" (UID: "242754b4-5708-477d-8069-01f034970ad7"). InnerVolumeSpecName "kube-api-access-ks79v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:47:28.508960 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:28.508873 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/242754b4-5708-477d-8069-01f034970ad7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "242754b4-5708-477d-8069-01f034970ad7" (UID: "242754b4-5708-477d-8069-01f034970ad7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 22:47:28.607912 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:28.607875 2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/242754b4-5708-477d-8069-01f034970ad7-maas-api-tls\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:47:28.607912 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:28.607904 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks79v\" (UniqueName: \"kubernetes.io/projected/242754b4-5708-477d-8069-01f034970ad7-kube-api-access-ks79v\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:47:28.607912 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:28.607915 2553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/242754b4-5708-477d-8069-01f034970ad7-openshift-service-ca\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:47:29.265327 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:29.265294 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72fe47-8bfb7984f-6bhd9" Jul 28 22:47:29.265327 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:29.265316 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72fe47-8bfb7984f-6bhd9" event={"ID":"242754b4-5708-477d-8069-01f034970ad7","Type":"ContainerDied","Data":"04acf731823cbec71ec37bbcd75573ce7334e4c2dcc03696928c81f798c5118e"} Jul 28 22:47:29.265862 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:29.265405 2553 scope.go:117] "RemoveContainer" containerID="5ee1830e588353e2ee02028c7dff17be4a6be01fb8fbcdc5ecfc7bfe17cd8614" Jul 28 22:47:29.285293 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:29.285268 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-72fe47-8bfb7984f-6bhd9"] Jul 28 22:47:29.289283 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:29.289262 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-72fe47-8bfb7984f-6bhd9"] Jul 28 22:47:30.405368 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:30.405337 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="242754b4-5708-477d-8069-01f034970ad7" path="/var/lib/kubelet/pods/242754b4-5708-477d-8069-01f034970ad7/volumes" Jul 28 22:47:34.856471 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:34.856439 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr"] Jul 28 22:47:34.856993 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:34.856653 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr" podUID="049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb" containerName="istio-proxy" containerID="cri-o://ca24a2484164b11f86257c816f89edb32c8e97066b937573ffc5f69a7a9aeca9" gracePeriod=30 Jul 28 22:47:35.726017 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:35.725985 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j"] Jul 28 22:47:35.726339 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:35.726327 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83e80fde-c25b-410e-9523-281a4a0450c5" containerName="maas-api" Jul 28 22:47:35.726339 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:35.726340 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="83e80fde-c25b-410e-9523-281a4a0450c5" containerName="maas-api" Jul 28 22:47:35.726431 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:35.726348 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4512921a-6fa4-4475-bc64-85169d264b10" containerName="revoke-keys" Jul 28 22:47:35.726431 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:35.726353 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="4512921a-6fa4-4475-bc64-85169d264b10" containerName="revoke-keys" Jul 28 22:47:35.726431 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:35.726363 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89223ac8-2798-48fe-b8c8-5e255b7b0bf2" containerName="istio-proxy" Jul 28 22:47:35.726431 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:35.726369 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="89223ac8-2798-48fe-b8c8-5e255b7b0bf2" containerName="istio-proxy" Jul 28 22:47:35.726431 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:35.726386 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="242754b4-5708-477d-8069-01f034970ad7" containerName="maas-api" Jul 28 22:47:35.726431 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:35.726391 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="242754b4-5708-477d-8069-01f034970ad7" containerName="maas-api" Jul 28 22:47:35.726616 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:35.726447 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="242754b4-5708-477d-8069-01f034970ad7" containerName="maas-api" Jul 28 22:47:35.726616 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:35.726455 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="89223ac8-2798-48fe-b8c8-5e255b7b0bf2" containerName="istio-proxy" Jul 28 22:47:35.726616 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:35.726463 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="83e80fde-c25b-410e-9523-281a4a0450c5" containerName="maas-api" Jul 28 22:47:35.726616 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:35.726471 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="4512921a-6fa4-4475-bc64-85169d264b10" containerName="revoke-keys" Jul 28 22:47:35.726616 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:35.726477 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="89223ac8-2798-48fe-b8c8-5e255b7b0bf2" containerName="istio-proxy" Jul 28 22:47:35.731071 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:35.731050 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j" Jul 28 22:47:35.733116 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:35.733097 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-63f77a-openshift-default-dockercfg-j6zq4\"" Jul 28 22:47:35.740649 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:35.740628 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j"] Jul 28 22:47:35.878034 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:35.878001 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a31e333e-b17f-44e6-bedc-612fcf103228-workload-socket\") pod \"e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j\" (UID: \"a31e333e-b17f-44e6-bedc-612fcf103228\") " pod="openshift-ingress/e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j" Jul 28 22:47:35.878034 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:35.878038 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a31e333e-b17f-44e6-bedc-612fcf103228-istio-data\") pod \"e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j\" (UID: \"a31e333e-b17f-44e6-bedc-612fcf103228\") " pod="openshift-ingress/e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j" Jul 28 22:47:35.878511 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:35.878072 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a31e333e-b17f-44e6-bedc-612fcf103228-credential-socket\") pod \"e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j\" (UID: \"a31e333e-b17f-44e6-bedc-612fcf103228\") " pod="openshift-ingress/e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j" Jul 28 22:47:35.878511 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:35.878198 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw6gv\" (UniqueName: \"kubernetes.io/projected/a31e333e-b17f-44e6-bedc-612fcf103228-kube-api-access-gw6gv\") pod \"e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j\" (UID: \"a31e333e-b17f-44e6-bedc-612fcf103228\") " pod="openshift-ingress/e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j" Jul 28 22:47:35.878511 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:35.878259 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a31e333e-b17f-44e6-bedc-612fcf103228-workload-certs\") pod \"e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j\" (UID: \"a31e333e-b17f-44e6-bedc-612fcf103228\") " pod="openshift-ingress/e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j" Jul 28 22:47:35.878511 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:35.878298 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a31e333e-b17f-44e6-bedc-612fcf103228-istio-token\") pod \"e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j\" (UID: \"a31e333e-b17f-44e6-bedc-612fcf103228\") " pod="openshift-ingress/e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j" Jul 28 22:47:35.878511 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:35.878344 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a31e333e-b17f-44e6-bedc-612fcf103228-istio-envoy\") pod \"e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j\" (UID: \"a31e333e-b17f-44e6-bedc-612fcf103228\") " pod="openshift-ingress/e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j" Jul 28 22:47:35.878511 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:35.878399 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a31e333e-b17f-44e6-bedc-612fcf103228-istio-podinfo\") pod \"e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j\" (UID: \"a31e333e-b17f-44e6-bedc-612fcf103228\") " pod="openshift-ingress/e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j" Jul 28 22:47:35.878511 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:35.878427 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a31e333e-b17f-44e6-bedc-612fcf103228-istiod-ca-cert\") pod \"e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j\" (UID: \"a31e333e-b17f-44e6-bedc-612fcf103228\") " pod="openshift-ingress/e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j" Jul 28 22:47:35.979529 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:35.979414 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a31e333e-b17f-44e6-bedc-612fcf103228-istio-podinfo\") pod \"e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j\" (UID: \"a31e333e-b17f-44e6-bedc-612fcf103228\") " pod="openshift-ingress/e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j" Jul 28 22:47:35.979529 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:35.979467 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a31e333e-b17f-44e6-bedc-612fcf103228-istiod-ca-cert\") pod \"e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j\" (UID: \"a31e333e-b17f-44e6-bedc-612fcf103228\") " pod="openshift-ingress/e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j" Jul 28 22:47:35.979529 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:35.979507 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a31e333e-b17f-44e6-bedc-612fcf103228-workload-socket\") pod \"e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j\" (UID: \"a31e333e-b17f-44e6-bedc-612fcf103228\") " pod="openshift-ingress/e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j" Jul 28 22:47:35.979860 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:35.979529 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a31e333e-b17f-44e6-bedc-612fcf103228-istio-data\") pod \"e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j\" (UID: \"a31e333e-b17f-44e6-bedc-612fcf103228\") " pod="openshift-ingress/e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j" Jul 28 22:47:35.979860 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:35.979680 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a31e333e-b17f-44e6-bedc-612fcf103228-credential-socket\") pod \"e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j\" (UID: \"a31e333e-b17f-44e6-bedc-612fcf103228\") " pod="openshift-ingress/e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j" Jul 28 22:47:35.979860 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:35.979781 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw6gv\" (UniqueName: \"kubernetes.io/projected/a31e333e-b17f-44e6-bedc-612fcf103228-kube-api-access-gw6gv\") pod \"e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j\" (UID: \"a31e333e-b17f-44e6-bedc-612fcf103228\") " pod="openshift-ingress/e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j" Jul 28 22:47:35.979860 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:35.979816 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a31e333e-b17f-44e6-bedc-612fcf103228-workload-certs\") pod \"e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j\" (UID: \"a31e333e-b17f-44e6-bedc-612fcf103228\") " pod="openshift-ingress/e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j" Jul 28 22:47:35.979860 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:35.979847 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a31e333e-b17f-44e6-bedc-612fcf103228-istio-token\") pod \"e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j\" (UID: \"a31e333e-b17f-44e6-bedc-612fcf103228\") " pod="openshift-ingress/e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j" Jul 28 22:47:35.980124 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:35.979879 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a31e333e-b17f-44e6-bedc-612fcf103228-istio-envoy\") pod \"e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j\" (UID: \"a31e333e-b17f-44e6-bedc-612fcf103228\") " pod="openshift-ingress/e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j" Jul 28 22:47:35.980124 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:35.979989 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a31e333e-b17f-44e6-bedc-612fcf103228-istio-data\") pod \"e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j\" (UID: \"a31e333e-b17f-44e6-bedc-612fcf103228\") " pod="openshift-ingress/e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j" Jul 28 22:47:35.980124 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:35.980046 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a31e333e-b17f-44e6-bedc-612fcf103228-workload-socket\") pod \"e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j\" (UID: \"a31e333e-b17f-44e6-bedc-612fcf103228\") " pod="openshift-ingress/e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j" Jul 28 22:47:35.980276 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:35.980254 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a31e333e-b17f-44e6-bedc-612fcf103228-workload-certs\") pod \"e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j\" (UID: \"a31e333e-b17f-44e6-bedc-612fcf103228\") " pod="openshift-ingress/e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j" Jul 28 22:47:35.980484 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:35.980460 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a31e333e-b17f-44e6-bedc-612fcf103228-istiod-ca-cert\") pod \"e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j\" (UID: \"a31e333e-b17f-44e6-bedc-612fcf103228\") " pod="openshift-ingress/e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j" Jul 28 22:47:35.980573 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:35.980482 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a31e333e-b17f-44e6-bedc-612fcf103228-credential-socket\") pod \"e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j\" (UID: \"a31e333e-b17f-44e6-bedc-612fcf103228\") " pod="openshift-ingress/e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j" Jul 28 22:47:35.982140 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:35.982118 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a31e333e-b17f-44e6-bedc-612fcf103228-istio-podinfo\") pod \"e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j\" (UID: \"a31e333e-b17f-44e6-bedc-612fcf103228\") " pod="openshift-ingress/e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j" Jul 28 22:47:35.982370 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:35.982353 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a31e333e-b17f-44e6-bedc-612fcf103228-istio-envoy\") pod \"e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j\" (UID: \"a31e333e-b17f-44e6-bedc-612fcf103228\") " pod="openshift-ingress/e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j" Jul 28 22:47:35.987804 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:35.987777 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw6gv\" (UniqueName: \"kubernetes.io/projected/a31e333e-b17f-44e6-bedc-612fcf103228-kube-api-access-gw6gv\") pod \"e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j\" (UID: \"a31e333e-b17f-44e6-bedc-612fcf103228\") " pod="openshift-ingress/e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j" Jul 28 22:47:35.988411 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:35.988383 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a31e333e-b17f-44e6-bedc-612fcf103228-istio-token\") pod \"e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j\" (UID: \"a31e333e-b17f-44e6-bedc-612fcf103228\") " pod="openshift-ingress/e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j" Jul 28 22:47:36.042233 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:36.042195 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j" Jul 28 22:47:36.185140 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:36.185111 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j"] Jul 28 22:47:36.186833 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:47:36.186807 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda31e333e_b17f_44e6_bedc_612fcf103228.slice/crio-f2c40082e014a578dd424710a61bf0775058802f2057ab69ac2773390dd143c0 WatchSource:0}: Error finding container f2c40082e014a578dd424710a61bf0775058802f2057ab69ac2773390dd143c0: Status 404 returned error can't find the container with id f2c40082e014a578dd424710a61bf0775058802f2057ab69ac2773390dd143c0 Jul 28 22:47:36.189091 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:36.189050 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:47:36.189184 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:36.189141 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:47:36.189230 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:36.189183 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:47:36.307461 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:36.307423 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j" event={"ID":"a31e333e-b17f-44e6-bedc-612fcf103228","Type":"ContainerStarted","Data":"e88bae5a6bd563507c2661158c6a32d2e2b4dbc2276118f92e7f5082957d56da"} Jul 28 22:47:36.307658 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:36.307489 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j" event={"ID":"a31e333e-b17f-44e6-bedc-612fcf103228","Type":"ContainerStarted","Data":"f2c40082e014a578dd424710a61bf0775058802f2057ab69ac2773390dd143c0"} Jul 28 22:47:36.330439 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:36.330382 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j" podStartSLOduration=1.3303605219999999 podStartE2EDuration="1.330360522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:47:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:47:36.329497902 +0000 UTC m=+2276.416755975" watchObservedRunningTime="2026-07-28 22:47:36.330360522 +0000 UTC m=+2276.417618599" Jul 28 22:47:37.043072 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:37.043032 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j" Jul 28 22:47:37.048103 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:37.048079 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j" Jul 28 22:47:37.313054 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:37.312956 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j" Jul 28 22:47:37.314090 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:37.314065 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j" Jul 28 22:47:40.336662 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:40.336621 2553 generic.go:356] "Generic (PLEG): container finished" podID="049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb" containerID="ca24a2484164b11f86257c816f89edb32c8e97066b937573ffc5f69a7a9aeca9" exitCode=0 Jul 28 22:47:40.337107 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:40.336863 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr" event={"ID":"049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb","Type":"ContainerDied","Data":"ca24a2484164b11f86257c816f89edb32c8e97066b937573ffc5f69a7a9aeca9"} Jul 28 22:47:40.337107 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:40.336979 2553 scope.go:117] "RemoveContainer" containerID="d408f39f38502858d706553286e63b620504c89409a460b305fc5b89867e973a" Jul 28 22:47:40.524351 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:40.524322 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr" Jul 28 22:47:40.624225 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:40.624123 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb-credential-socket\") pod \"049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb\" (UID: \"049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb\") " Jul 28 22:47:40.624404 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:40.624229 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb-istio-data\") pod \"049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb\" (UID: \"049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb\") " Jul 28 22:47:40.624404 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:40.624256 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2mlw8\" (UniqueName: \"kubernetes.io/projected/049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb-kube-api-access-2mlw8\") pod \"049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb\" (UID: \"049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb\") " Jul 28 22:47:40.624404 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:40.624317 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb-istio-token\") pod \"049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb\" (UID: \"049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb\") " Jul 28 22:47:40.624404 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:40.624348 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb-istio-podinfo\") pod \"049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb\" (UID: \"049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb\") " Jul 28 22:47:40.624597 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:40.624436 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb" (UID: "049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:47:40.624789 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:40.624372 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb-istio-envoy\") pod \"049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb\" (UID: \"049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb\") " Jul 28 22:47:40.624935 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:40.624845 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb-workload-certs\") pod \"049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb\" (UID: \"049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb\") " Jul 28 22:47:40.625297 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:40.625152 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb" (UID: "049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:47:40.625397 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:40.625376 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb" (UID: "049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:47:40.625472 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:40.625452 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb-istio-data" (OuterVolumeSpecName: "istio-data") pod "049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb" (UID: "049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:47:40.625542 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:40.625460 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb-workload-socket\") pod \"049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb\" (UID: \"049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb\") " Jul 28 22:47:40.625602 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:40.625535 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb-istiod-ca-cert\") pod \"049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb\" (UID: \"049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb\") " Jul 28 22:47:40.625845 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:40.625824 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb" (UID: "049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 22:47:40.625955 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:40.625905 2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb-credential-socket\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:47:40.625955 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:40.625921 2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb-istio-data\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:47:40.625955 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:40.625935 2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb-workload-certs\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:47:40.625955 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:40.625946 2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb-workload-socket\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:47:40.626161 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:40.625959 2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb-istiod-ca-cert\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:47:40.627048 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:40.627020 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb" (UID: "049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:47:40.627168 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:40.627087 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb-istio-token" (OuterVolumeSpecName: "istio-token") pod "049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb" (UID: "049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:47:40.627233 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:40.627202 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb" (UID: "049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 22:47:40.627233 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:40.627200 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb-kube-api-access-2mlw8" (OuterVolumeSpecName: "kube-api-access-2mlw8") pod "049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb" (UID: "049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb"). InnerVolumeSpecName "kube-api-access-2mlw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:47:40.726408 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:40.726368 2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb-istio-token\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:47:40.726408 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:40.726403 2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb-istio-podinfo\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:47:40.726408 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:40.726414 2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb-istio-envoy\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:47:40.726641 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:40.726426 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mlw8\" (UniqueName: \"kubernetes.io/projected/049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb-kube-api-access-2mlw8\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:47:41.344027 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:41.344000 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr" Jul 28 22:47:41.344468 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:41.344026 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr" event={"ID":"049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb","Type":"ContainerDied","Data":"3dc36ed2181b9ffa09c8b08342979fde7e263e8997d3e4346d6ba4b778ed3a06"} Jul 28 22:47:41.344468 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:41.344086 2553 scope.go:117] "RemoveContainer" containerID="ca24a2484164b11f86257c816f89edb32c8e97066b937573ffc5f69a7a9aeca9" Jul 28 22:47:41.365982 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:41.365954 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr"] Jul 28 22:47:41.373538 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:41.373515 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-72fe47-openshift-default-6588b847c4-pk6gr"] Jul 28 22:47:42.404927 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:42.404897 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb" path="/var/lib/kubelet/pods/049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb/volumes" Jul 28 22:47:52.314211 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:52.314175 2553 patch_prober.go:28] interesting pod/e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 28 22:47:52.314646 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:52.314235 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j" podUID="a31e333e-b17f-44e6-bedc-612fcf103228" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 28 22:47:52.407833 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:52.407795 2553 generic.go:356] "Generic (PLEG): container finished" podID="a31e333e-b17f-44e6-bedc-612fcf103228" containerID="e88bae5a6bd563507c2661158c6a32d2e2b4dbc2276118f92e7f5082957d56da" exitCode=0 Jul 28 22:47:52.408006 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:52.407866 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j" event={"ID":"a31e333e-b17f-44e6-bedc-612fcf103228","Type":"ContainerDied","Data":"e88bae5a6bd563507c2661158c6a32d2e2b4dbc2276118f92e7f5082957d56da"} Jul 28 22:47:52.408232 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:52.408218 2553 scope.go:117] "RemoveContainer" containerID="e88bae5a6bd563507c2661158c6a32d2e2b4dbc2276118f92e7f5082957d56da" Jul 28 22:47:52.409227 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:52.409201 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:47:52.409326 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:52.409263 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:47:52.409326 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:52.409295 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:47:53.420148 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:53.420106 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j" event={"ID":"a31e333e-b17f-44e6-bedc-612fcf103228","Type":"ContainerStarted","Data":"b7969a64e90bc0074b68d4d9b666f118667df8a66c6688205462464452de0f57"} Jul 28 22:47:54.043113 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:54.043075 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j" Jul 28 22:47:54.043373 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:54.043350 2553 patch_prober.go:28] interesting pod/e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 28 22:47:54.043421 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:54.043401 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j" podUID="a31e333e-b17f-44e6-bedc-612fcf103228" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 28 22:47:55.042859 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:55.042818 2553 patch_prober.go:28] interesting pod/e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 28 22:47:55.043289 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:55.042888 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j" podUID="a31e333e-b17f-44e6-bedc-612fcf103228" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 28 22:47:56.043680 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:56.043642 2553 patch_prober.go:28] interesting pod/e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 28 22:47:56.044127 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:56.043727 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j" podUID="a31e333e-b17f-44e6-bedc-612fcf103228" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 28 22:47:57.043061 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:57.043024 2553 patch_prober.go:28] interesting pod/e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 28 22:47:57.043236 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:57.043080 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j" podUID="a31e333e-b17f-44e6-bedc-612fcf103228" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 28 22:47:58.046955 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:58.046921 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j" Jul 28 22:47:58.047373 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:58.047295 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j" Jul 28 22:47:58.048051 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:47:58.048032 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j" Jul 28 22:48:04.039995 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:04.039960 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9"] Jul 28 22:48:04.040417 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:04.040356 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89223ac8-2798-48fe-b8c8-5e255b7b0bf2" containerName="istio-proxy" Jul 28 22:48:04.040417 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:04.040373 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="89223ac8-2798-48fe-b8c8-5e255b7b0bf2" containerName="istio-proxy" Jul 28 22:48:04.040417 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:04.040386 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb" containerName="istio-proxy" Jul 28 22:48:04.040417 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:04.040392 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb" containerName="istio-proxy" Jul 28 22:48:04.040417 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:04.040405 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb" containerName="istio-proxy" Jul 28 22:48:04.040417 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:04.040410 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb" containerName="istio-proxy" Jul 28 22:48:04.040633 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:04.040472 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb" containerName="istio-proxy" Jul 28 22:48:04.040633 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:04.040483 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="049c2f5a-16ba-48b4-b6f3-6f3ae4bcbedb" containerName="istio-proxy" Jul 28 22:48:04.043750 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:04.043725 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9" Jul 28 22:48:04.045837 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:04.045814 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-472fa3-openshift-default-dockercfg-89zcl\"" Jul 28 22:48:04.054352 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:04.054327 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9"] Jul 28 22:48:04.136196 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:04.136150 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11ea77c8-026d-45c0-8023-f2d4a965666b-istio-envoy\") pod \"e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9\" (UID: \"11ea77c8-026d-45c0-8023-f2d4a965666b\") " pod="openshift-ingress/e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9" Jul 28 22:48:04.136394 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:04.136206 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hffq\" (UniqueName: \"kubernetes.io/projected/11ea77c8-026d-45c0-8023-f2d4a965666b-kube-api-access-8hffq\") pod \"e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9\" (UID: \"11ea77c8-026d-45c0-8023-f2d4a965666b\") " pod="openshift-ingress/e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9" Jul 28 22:48:04.136394 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:04.136279 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11ea77c8-026d-45c0-8023-f2d4a965666b-istio-data\") pod \"e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9\" (UID: \"11ea77c8-026d-45c0-8023-f2d4a965666b\") " pod="openshift-ingress/e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9" Jul 28 22:48:04.136394 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:04.136334 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11ea77c8-026d-45c0-8023-f2d4a965666b-workload-certs\") pod \"e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9\" (UID: \"11ea77c8-026d-45c0-8023-f2d4a965666b\") " pod="openshift-ingress/e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9" Jul 28 22:48:04.136394 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:04.136386 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ea77c8-026d-45c0-8023-f2d4a965666b-credential-socket\") pod \"e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9\" (UID: \"11ea77c8-026d-45c0-8023-f2d4a965666b\") " pod="openshift-ingress/e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9" Jul 28 22:48:04.136640 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:04.136428 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11ea77c8-026d-45c0-8023-f2d4a965666b-istio-podinfo\") pod \"e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9\" (UID: \"11ea77c8-026d-45c0-8023-f2d4a965666b\") " pod="openshift-ingress/e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9" Jul 28 22:48:04.136640 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:04.136456 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11ea77c8-026d-45c0-8023-f2d4a965666b-istiod-ca-cert\") pod \"e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9\" (UID: \"11ea77c8-026d-45c0-8023-f2d4a965666b\") " pod="openshift-ingress/e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9" Jul 28 22:48:04.136640 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:04.136573 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11ea77c8-026d-45c0-8023-f2d4a965666b-istio-token\") pod \"e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9\" (UID: \"11ea77c8-026d-45c0-8023-f2d4a965666b\") " pod="openshift-ingress/e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9" Jul 28 22:48:04.136640 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:04.136601 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ea77c8-026d-45c0-8023-f2d4a965666b-workload-socket\") pod \"e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9\" (UID: \"11ea77c8-026d-45c0-8023-f2d4a965666b\") " pod="openshift-ingress/e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9" Jul 28 22:48:04.237102 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:04.237065 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11ea77c8-026d-45c0-8023-f2d4a965666b-istio-token\") pod \"e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9\" (UID: \"11ea77c8-026d-45c0-8023-f2d4a965666b\") " pod="openshift-ingress/e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9" Jul 28 22:48:04.237102 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:04.237103 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ea77c8-026d-45c0-8023-f2d4a965666b-workload-socket\") pod \"e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9\" (UID: \"11ea77c8-026d-45c0-8023-f2d4a965666b\") " pod="openshift-ingress/e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9" Jul 28 22:48:04.237359 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:04.237129 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11ea77c8-026d-45c0-8023-f2d4a965666b-istio-envoy\") pod \"e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9\" (UID: \"11ea77c8-026d-45c0-8023-f2d4a965666b\") " pod="openshift-ingress/e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9" Jul 28 22:48:04.237359 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:04.237147 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hffq\" (UniqueName: \"kubernetes.io/projected/11ea77c8-026d-45c0-8023-f2d4a965666b-kube-api-access-8hffq\") pod \"e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9\" (UID: \"11ea77c8-026d-45c0-8023-f2d4a965666b\") " pod="openshift-ingress/e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9" Jul 28 22:48:04.237359 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:04.237168 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11ea77c8-026d-45c0-8023-f2d4a965666b-istio-data\") pod \"e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9\" (UID: \"11ea77c8-026d-45c0-8023-f2d4a965666b\") " pod="openshift-ingress/e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9" Jul 28 22:48:04.237359 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:04.237186 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11ea77c8-026d-45c0-8023-f2d4a965666b-workload-certs\") pod \"e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9\" (UID: \"11ea77c8-026d-45c0-8023-f2d4a965666b\") " pod="openshift-ingress/e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9" Jul 28 22:48:04.237359 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:04.237222 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ea77c8-026d-45c0-8023-f2d4a965666b-credential-socket\") pod \"e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9\" (UID: \"11ea77c8-026d-45c0-8023-f2d4a965666b\") " pod="openshift-ingress/e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9" Jul 28 22:48:04.237359 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:04.237252 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11ea77c8-026d-45c0-8023-f2d4a965666b-istio-podinfo\") pod \"e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9\" (UID: \"11ea77c8-026d-45c0-8023-f2d4a965666b\") " pod="openshift-ingress/e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9" Jul 28 22:48:04.237359 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:04.237272 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11ea77c8-026d-45c0-8023-f2d4a965666b-istiod-ca-cert\") pod \"e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9\" (UID: \"11ea77c8-026d-45c0-8023-f2d4a965666b\") " pod="openshift-ingress/e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9" Jul 28 22:48:04.237737 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:04.237637 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11ea77c8-026d-45c0-8023-f2d4a965666b-istio-data\") pod \"e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9\" (UID: \"11ea77c8-026d-45c0-8023-f2d4a965666b\") " pod="openshift-ingress/e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9" Jul 28 22:48:04.237803 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:04.237748 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ea77c8-026d-45c0-8023-f2d4a965666b-credential-socket\") pod \"e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9\" (UID: \"11ea77c8-026d-45c0-8023-f2d4a965666b\") " pod="openshift-ingress/e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9" Jul 28 22:48:04.237803 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:04.237779 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11ea77c8-026d-45c0-8023-f2d4a965666b-workload-certs\") pod \"e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9\" (UID: \"11ea77c8-026d-45c0-8023-f2d4a965666b\") " pod="openshift-ingress/e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9" Jul 28 22:48:04.238082 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:04.237827 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ea77c8-026d-45c0-8023-f2d4a965666b-workload-socket\") pod \"e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9\" (UID: \"11ea77c8-026d-45c0-8023-f2d4a965666b\") " pod="openshift-ingress/e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9" Jul 28 22:48:04.238082 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:04.237993 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11ea77c8-026d-45c0-8023-f2d4a965666b-istiod-ca-cert\") pod \"e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9\" (UID: \"11ea77c8-026d-45c0-8023-f2d4a965666b\") " pod="openshift-ingress/e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9" Jul 28 22:48:04.239652 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:04.239629 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11ea77c8-026d-45c0-8023-f2d4a965666b-istio-envoy\") pod \"e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9\" (UID: \"11ea77c8-026d-45c0-8023-f2d4a965666b\") " pod="openshift-ingress/e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9" Jul 28 22:48:04.239845 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:04.239830 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11ea77c8-026d-45c0-8023-f2d4a965666b-istio-podinfo\") pod \"e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9\" (UID: \"11ea77c8-026d-45c0-8023-f2d4a965666b\") " pod="openshift-ingress/e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9" Jul 28 22:48:04.244668 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:04.244643 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hffq\" (UniqueName: \"kubernetes.io/projected/11ea77c8-026d-45c0-8023-f2d4a965666b-kube-api-access-8hffq\") pod \"e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9\" (UID: \"11ea77c8-026d-45c0-8023-f2d4a965666b\") " pod="openshift-ingress/e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9" Jul 28 22:48:04.245268 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:04.245250 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11ea77c8-026d-45c0-8023-f2d4a965666b-istio-token\") pod \"e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9\" (UID: \"11ea77c8-026d-45c0-8023-f2d4a965666b\") " pod="openshift-ingress/e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9" Jul 28 22:48:04.355918 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:04.355816 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9" Jul 28 22:48:04.524273 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:04.524245 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9"] Jul 28 22:48:04.525851 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:48:04.525821 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11ea77c8_026d_45c0_8023_f2d4a965666b.slice/crio-cb81ffcc9ff136f9667c1ea0d351fa6b2be86718202453692baef1a0d2a12ee8 WatchSource:0}: Error finding container cb81ffcc9ff136f9667c1ea0d351fa6b2be86718202453692baef1a0d2a12ee8: Status 404 returned error can't find the container with id cb81ffcc9ff136f9667c1ea0d351fa6b2be86718202453692baef1a0d2a12ee8 Jul 28 22:48:04.528044 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:04.528014 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:48:04.528117 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:04.528080 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:48:04.528117 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:04.528108 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:48:05.496742 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:05.496690 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9" event={"ID":"11ea77c8-026d-45c0-8023-f2d4a965666b","Type":"ContainerStarted","Data":"39431c32e51f72ba04a7657c3e9fcb49746a1c1fe138779f8ca46db96d3a9564"} Jul 28 22:48:05.497198 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:05.496764 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9" event={"ID":"11ea77c8-026d-45c0-8023-f2d4a965666b","Type":"ContainerStarted","Data":"cb81ffcc9ff136f9667c1ea0d351fa6b2be86718202453692baef1a0d2a12ee8"} Jul 28 22:48:05.516779 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:05.516726 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9" podStartSLOduration=1.516691789 podStartE2EDuration="1.516691789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:48:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:48:05.514214234 +0000 UTC m=+2305.601472302" watchObservedRunningTime="2026-07-28 22:48:05.516691789 +0000 UTC m=+2305.603949930" Jul 28 22:48:06.356475 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:06.356439 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9" Jul 28 22:48:06.361555 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:06.361527 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9" Jul 28 22:48:06.503360 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:06.503326 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9" Jul 28 22:48:06.504266 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:06.504249 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9" Jul 28 22:48:07.597347 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:07.597315 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-472fa3-5466868976-jv2fk"] Jul 28 22:48:07.600924 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:07.600904 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-472fa3-5466868976-jv2fk" Jul 28 22:48:07.602883 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:07.602861 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-472fa3-serving-cert\"" Jul 28 22:48:07.610894 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:07.610871 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-472fa3-5466868976-jv2fk"] Jul 28 22:48:07.667072 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:07.667040 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfzhn\" (UniqueName: \"kubernetes.io/projected/625eb68e-7c1b-4758-8e74-2d2052645fff-kube-api-access-wfzhn\") pod \"maas-api-e2e-api-b-472fa3-5466868976-jv2fk\" (UID: \"625eb68e-7c1b-4758-8e74-2d2052645fff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-472fa3-5466868976-jv2fk" Jul 28 22:48:07.667248 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:07.667112 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/625eb68e-7c1b-4758-8e74-2d2052645fff-openshift-service-ca\") pod \"maas-api-e2e-api-b-472fa3-5466868976-jv2fk\" (UID: \"625eb68e-7c1b-4758-8e74-2d2052645fff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-472fa3-5466868976-jv2fk" Jul 28 22:48:07.667248 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:07.667173 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/625eb68e-7c1b-4758-8e74-2d2052645fff-maas-api-tls\") pod \"maas-api-e2e-api-b-472fa3-5466868976-jv2fk\" (UID: \"625eb68e-7c1b-4758-8e74-2d2052645fff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-472fa3-5466868976-jv2fk" Jul 28 22:48:07.768406 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:07.768366 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/625eb68e-7c1b-4758-8e74-2d2052645fff-openshift-service-ca\") pod \"maas-api-e2e-api-b-472fa3-5466868976-jv2fk\" (UID: \"625eb68e-7c1b-4758-8e74-2d2052645fff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-472fa3-5466868976-jv2fk" Jul 28 22:48:07.768588 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:07.768421 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/625eb68e-7c1b-4758-8e74-2d2052645fff-maas-api-tls\") pod \"maas-api-e2e-api-b-472fa3-5466868976-jv2fk\" (UID: \"625eb68e-7c1b-4758-8e74-2d2052645fff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-472fa3-5466868976-jv2fk" Jul 28 22:48:07.768588 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:07.768448 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfzhn\" (UniqueName: \"kubernetes.io/projected/625eb68e-7c1b-4758-8e74-2d2052645fff-kube-api-access-wfzhn\") pod \"maas-api-e2e-api-b-472fa3-5466868976-jv2fk\" (UID: \"625eb68e-7c1b-4758-8e74-2d2052645fff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-472fa3-5466868976-jv2fk" Jul 28 22:48:07.768588 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:48:07.768534 2553 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-472fa3-serving-cert: secret "maas-api-e2e-api-b-472fa3-serving-cert" not found Jul 28 22:48:07.768773 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:48:07.768659 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/625eb68e-7c1b-4758-8e74-2d2052645fff-maas-api-tls podName:625eb68e-7c1b-4758-8e74-2d2052645fff nodeName:}" failed. No retries permitted until 2026-07-28 22:48:08.268635875 +0000 UTC m=+2308.355893939 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/625eb68e-7c1b-4758-8e74-2d2052645fff-maas-api-tls") pod "maas-api-e2e-api-b-472fa3-5466868976-jv2fk" (UID: "625eb68e-7c1b-4758-8e74-2d2052645fff") : secret "maas-api-e2e-api-b-472fa3-serving-cert" not found Jul 28 22:48:07.769101 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:07.769077 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/625eb68e-7c1b-4758-8e74-2d2052645fff-openshift-service-ca\") pod \"maas-api-e2e-api-b-472fa3-5466868976-jv2fk\" (UID: \"625eb68e-7c1b-4758-8e74-2d2052645fff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-472fa3-5466868976-jv2fk" Jul 28 22:48:07.778750 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:07.778724 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfzhn\" (UniqueName: \"kubernetes.io/projected/625eb68e-7c1b-4758-8e74-2d2052645fff-kube-api-access-wfzhn\") pod \"maas-api-e2e-api-b-472fa3-5466868976-jv2fk\" (UID: \"625eb68e-7c1b-4758-8e74-2d2052645fff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-472fa3-5466868976-jv2fk" Jul 28 22:48:08.273128 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:08.273091 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/625eb68e-7c1b-4758-8e74-2d2052645fff-maas-api-tls\") pod \"maas-api-e2e-api-b-472fa3-5466868976-jv2fk\" (UID: \"625eb68e-7c1b-4758-8e74-2d2052645fff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-472fa3-5466868976-jv2fk" Jul 28 22:48:08.275872 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:08.275844 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/625eb68e-7c1b-4758-8e74-2d2052645fff-maas-api-tls\") pod \"maas-api-e2e-api-b-472fa3-5466868976-jv2fk\" (UID: \"625eb68e-7c1b-4758-8e74-2d2052645fff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-472fa3-5466868976-jv2fk" Jul 28 22:48:08.513933 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:08.513904 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-472fa3-5466868976-jv2fk" Jul 28 22:48:08.652679 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:08.652655 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-472fa3-5466868976-jv2fk"] Jul 28 22:48:08.654765 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:48:08.654737 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod625eb68e_7c1b_4758_8e74_2d2052645fff.slice/crio-d521f1735c18b6662b7b7b133b5bf29286aae4f8b582a908a596d8dbb0c3614c WatchSource:0}: Error finding container d521f1735c18b6662b7b7b133b5bf29286aae4f8b582a908a596d8dbb0c3614c: Status 404 returned error can't find the container with id d521f1735c18b6662b7b7b133b5bf29286aae4f8b582a908a596d8dbb0c3614c Jul 28 22:48:08.656215 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:08.656195 2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 22:48:09.521785 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:09.521736 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-472fa3-5466868976-jv2fk" event={"ID":"625eb68e-7c1b-4758-8e74-2d2052645fff","Type":"ContainerStarted","Data":"729c41751b2163a0aef21b8812f89a8094ac0001105b52abe2a7cdf8fd5a4170"} Jul 28 22:48:09.521982 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:09.521794 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-472fa3-5466868976-jv2fk" event={"ID":"625eb68e-7c1b-4758-8e74-2d2052645fff","Type":"ContainerStarted","Data":"d521f1735c18b6662b7b7b133b5bf29286aae4f8b582a908a596d8dbb0c3614c"} Jul 28 22:48:09.521982 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:09.521855 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-472fa3-5466868976-jv2fk" Jul 28 22:48:09.540907 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:09.540857 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-472fa3-5466868976-jv2fk" podStartSLOduration=2.148261798 podStartE2EDuration="2.540841887s" totalImagesPullingTime="392.580089ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:48:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:48:09.539016714 +0000 UTC m=+2309.626274803" watchObservedRunningTime="2026-07-28 22:48:09.540841887 +0000 UTC m=+2309.628099955" Jul 28 22:48:15.539602 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:15.539566 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-472fa3-5466868976-jv2fk" Jul 28 22:48:22.469070 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:22.468527 2553 patch_prober.go:28] interesting pod/e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": read tcp 10.132.0.2:52134->10.132.0.64:15021: read: connection reset by peer" start-of-body= Jul 28 22:48:22.469070 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:22.468592 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9" podUID="11ea77c8-026d-45c0-8023-f2d4a965666b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": read tcp 10.132.0.2:52134->10.132.0.64:15021: read: connection reset by peer" Jul 28 22:48:22.607641 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:22.607598 2553 generic.go:356] "Generic (PLEG): container finished" podID="11ea77c8-026d-45c0-8023-f2d4a965666b" containerID="39431c32e51f72ba04a7657c3e9fcb49746a1c1fe138779f8ca46db96d3a9564" exitCode=0 Jul 28 22:48:22.607859 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:22.607672 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9" event={"ID":"11ea77c8-026d-45c0-8023-f2d4a965666b","Type":"ContainerDied","Data":"39431c32e51f72ba04a7657c3e9fcb49746a1c1fe138779f8ca46db96d3a9564"} Jul 28 22:48:22.608145 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:22.608127 2553 scope.go:117] "RemoveContainer" containerID="39431c32e51f72ba04a7657c3e9fcb49746a1c1fe138779f8ca46db96d3a9564" Jul 28 22:48:22.609117 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:22.609091 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:48:22.609206 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:22.609155 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:48:22.609206 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:22.609187 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:48:23.618771 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:23.617824 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9" event={"ID":"11ea77c8-026d-45c0-8023-f2d4a965666b","Type":"ContainerStarted","Data":"3d05e7be2d0bc47104fee2651858d5f7784948c3cd2ce69fa17e52bd283e0295"} Jul 28 22:48:24.355978 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:24.355936 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9" Jul 28 22:48:24.356246 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:24.356223 2553 patch_prober.go:28] interesting pod/e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 28 22:48:24.356310 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:24.356271 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9" podUID="11ea77c8-026d-45c0-8023-f2d4a965666b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 28 22:48:25.356546 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:25.356512 2553 patch_prober.go:28] interesting pod/e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 28 22:48:25.356971 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:25.356565 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9" podUID="11ea77c8-026d-45c0-8023-f2d4a965666b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 28 22:48:26.357232 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:26.357192 2553 patch_prober.go:28] interesting pod/e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 28 22:48:26.357637 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:26.357248 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9" podUID="11ea77c8-026d-45c0-8023-f2d4a965666b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 28 22:48:27.357075 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:27.357038 2553 patch_prober.go:28] interesting pod/e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 28 22:48:27.357268 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:27.357108 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9" podUID="11ea77c8-026d-45c0-8023-f2d4a965666b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 28 22:48:28.359632 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:28.359604 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9" Jul 28 22:48:28.360028 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:28.359967 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9" Jul 28 22:48:28.360678 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:28.360656 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9" Jul 28 22:48:52.863099 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:52.863060 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j"] Jul 28 22:48:52.863623 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:52.863361 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j" podUID="a31e333e-b17f-44e6-bedc-612fcf103228" containerName="istio-proxy" containerID="cri-o://b7969a64e90bc0074b68d4d9b666f118667df8a66c6688205462464452de0f57" gracePeriod=30 Jul 28 22:48:58.125262 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:58.125237 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j" Jul 28 22:48:58.232582 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:58.232547 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a31e333e-b17f-44e6-bedc-612fcf103228-credential-socket\") pod \"a31e333e-b17f-44e6-bedc-612fcf103228\" (UID: \"a31e333e-b17f-44e6-bedc-612fcf103228\") " Jul 28 22:48:58.232801 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:58.232613 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a31e333e-b17f-44e6-bedc-612fcf103228-istio-data\") pod \"a31e333e-b17f-44e6-bedc-612fcf103228\" (UID: \"a31e333e-b17f-44e6-bedc-612fcf103228\") " Jul 28 22:48:58.232801 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:58.232635 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a31e333e-b17f-44e6-bedc-612fcf103228-istio-envoy\") pod \"a31e333e-b17f-44e6-bedc-612fcf103228\" (UID: \"a31e333e-b17f-44e6-bedc-612fcf103228\") " Jul 28 22:48:58.232801 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:58.232679 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a31e333e-b17f-44e6-bedc-612fcf103228-istio-token\") pod \"a31e333e-b17f-44e6-bedc-612fcf103228\" (UID: \"a31e333e-b17f-44e6-bedc-612fcf103228\") " Jul 28 22:48:58.232801 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:58.232737 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a31e333e-b17f-44e6-bedc-612fcf103228-istiod-ca-cert\") pod \"a31e333e-b17f-44e6-bedc-612fcf103228\" (UID: \"a31e333e-b17f-44e6-bedc-612fcf103228\") " Jul 28 22:48:58.232801 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:58.232770 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a31e333e-b17f-44e6-bedc-612fcf103228-workload-socket\") pod \"a31e333e-b17f-44e6-bedc-612fcf103228\" (UID: \"a31e333e-b17f-44e6-bedc-612fcf103228\") " Jul 28 22:48:58.233095 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:58.232932 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a31e333e-b17f-44e6-bedc-612fcf103228-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a31e333e-b17f-44e6-bedc-612fcf103228" (UID: "a31e333e-b17f-44e6-bedc-612fcf103228"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:48:58.233095 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:58.233001 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gw6gv\" (UniqueName: \"kubernetes.io/projected/a31e333e-b17f-44e6-bedc-612fcf103228-kube-api-access-gw6gv\") pod \"a31e333e-b17f-44e6-bedc-612fcf103228\" (UID: \"a31e333e-b17f-44e6-bedc-612fcf103228\") " Jul 28 22:48:58.233095 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:58.233047 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a31e333e-b17f-44e6-bedc-612fcf103228-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a31e333e-b17f-44e6-bedc-612fcf103228" (UID: "a31e333e-b17f-44e6-bedc-612fcf103228"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:48:58.233095 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:58.233068 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a31e333e-b17f-44e6-bedc-612fcf103228-istio-podinfo\") pod \"a31e333e-b17f-44e6-bedc-612fcf103228\" (UID: \"a31e333e-b17f-44e6-bedc-612fcf103228\") " Jul 28 22:48:58.233313 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:58.233129 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a31e333e-b17f-44e6-bedc-612fcf103228-workload-certs\") pod \"a31e333e-b17f-44e6-bedc-612fcf103228\" (UID: \"a31e333e-b17f-44e6-bedc-612fcf103228\") " Jul 28 22:48:58.233313 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:58.233241 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a31e333e-b17f-44e6-bedc-612fcf103228-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a31e333e-b17f-44e6-bedc-612fcf103228" (UID: "a31e333e-b17f-44e6-bedc-612fcf103228"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 22:48:58.233419 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:58.233398 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a31e333e-b17f-44e6-bedc-612fcf103228-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a31e333e-b17f-44e6-bedc-612fcf103228" (UID: "a31e333e-b17f-44e6-bedc-612fcf103228"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:48:58.233588 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:58.233571 2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a31e333e-b17f-44e6-bedc-612fcf103228-credential-socket\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:48:58.233667 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:58.233594 2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a31e333e-b17f-44e6-bedc-612fcf103228-istiod-ca-cert\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:48:58.233667 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:58.233609 2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a31e333e-b17f-44e6-bedc-612fcf103228-workload-socket\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:48:58.233667 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:58.233621 2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a31e333e-b17f-44e6-bedc-612fcf103228-workload-certs\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:48:58.233839 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:58.233818 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a31e333e-b17f-44e6-bedc-612fcf103228-istio-data" (OuterVolumeSpecName: "istio-data") pod "a31e333e-b17f-44e6-bedc-612fcf103228" (UID: "a31e333e-b17f-44e6-bedc-612fcf103228"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:48:58.235222 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:58.235190 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a31e333e-b17f-44e6-bedc-612fcf103228-kube-api-access-gw6gv" (OuterVolumeSpecName: "kube-api-access-gw6gv") pod "a31e333e-b17f-44e6-bedc-612fcf103228" (UID: "a31e333e-b17f-44e6-bedc-612fcf103228"). InnerVolumeSpecName "kube-api-access-gw6gv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:48:58.235336 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:58.235249 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a31e333e-b17f-44e6-bedc-612fcf103228-istio-token" (OuterVolumeSpecName: "istio-token") pod "a31e333e-b17f-44e6-bedc-612fcf103228" (UID: "a31e333e-b17f-44e6-bedc-612fcf103228"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:48:58.235634 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:58.235605 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a31e333e-b17f-44e6-bedc-612fcf103228-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a31e333e-b17f-44e6-bedc-612fcf103228" (UID: "a31e333e-b17f-44e6-bedc-612fcf103228"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:48:58.235750 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:58.235638 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a31e333e-b17f-44e6-bedc-612fcf103228-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a31e333e-b17f-44e6-bedc-612fcf103228" (UID: "a31e333e-b17f-44e6-bedc-612fcf103228"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 22:48:58.334467 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:58.334435 2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a31e333e-b17f-44e6-bedc-612fcf103228-istio-data\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:48:58.334467 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:58.334461 2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a31e333e-b17f-44e6-bedc-612fcf103228-istio-envoy\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:48:58.334467 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:58.334470 2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a31e333e-b17f-44e6-bedc-612fcf103228-istio-token\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:48:58.334693 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:58.334479 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw6gv\" (UniqueName: \"kubernetes.io/projected/a31e333e-b17f-44e6-bedc-612fcf103228-kube-api-access-gw6gv\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:48:58.334693 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:58.334489 2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a31e333e-b17f-44e6-bedc-612fcf103228-istio-podinfo\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:48:58.825443 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:58.825409 2553 generic.go:356] "Generic (PLEG): container finished" podID="a31e333e-b17f-44e6-bedc-612fcf103228" containerID="b7969a64e90bc0074b68d4d9b666f118667df8a66c6688205462464452de0f57" exitCode=0 Jul 28 22:48:58.825617 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:58.825461 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j" Jul 28 22:48:58.825617 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:58.825498 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j" event={"ID":"a31e333e-b17f-44e6-bedc-612fcf103228","Type":"ContainerDied","Data":"b7969a64e90bc0074b68d4d9b666f118667df8a66c6688205462464452de0f57"} Jul 28 22:48:58.825617 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:58.825559 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j" event={"ID":"a31e333e-b17f-44e6-bedc-612fcf103228","Type":"ContainerDied","Data":"f2c40082e014a578dd424710a61bf0775058802f2057ab69ac2773390dd143c0"} Jul 28 22:48:58.825617 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:58.825589 2553 scope.go:117] "RemoveContainer" containerID="b7969a64e90bc0074b68d4d9b666f118667df8a66c6688205462464452de0f57" Jul 28 22:48:58.834872 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:58.834855 2553 scope.go:117] "RemoveContainer" containerID="e88bae5a6bd563507c2661158c6a32d2e2b4dbc2276118f92e7f5082957d56da" Jul 28 22:48:58.843107 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:58.843082 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j"] Jul 28 22:48:58.843970 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:58.843814 2553 scope.go:117] "RemoveContainer" containerID="b7969a64e90bc0074b68d4d9b666f118667df8a66c6688205462464452de0f57" Jul 28 22:48:58.844174 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:48:58.844144 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7969a64e90bc0074b68d4d9b666f118667df8a66c6688205462464452de0f57\": container with ID starting with b7969a64e90bc0074b68d4d9b666f118667df8a66c6688205462464452de0f57 not found: ID does not exist" containerID="b7969a64e90bc0074b68d4d9b666f118667df8a66c6688205462464452de0f57" Jul 28 22:48:58.844246 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:58.844187 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7969a64e90bc0074b68d4d9b666f118667df8a66c6688205462464452de0f57"} err="failed to get container status \"b7969a64e90bc0074b68d4d9b666f118667df8a66c6688205462464452de0f57\": rpc error: code = NotFound desc = could not find container \"b7969a64e90bc0074b68d4d9b666f118667df8a66c6688205462464452de0f57\": container with ID starting with b7969a64e90bc0074b68d4d9b666f118667df8a66c6688205462464452de0f57 not found: ID does not exist" Jul 28 22:48:58.844246 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:58.844212 2553 scope.go:117] "RemoveContainer" containerID="e88bae5a6bd563507c2661158c6a32d2e2b4dbc2276118f92e7f5082957d56da" Jul 28 22:48:58.844556 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:48:58.844517 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e88bae5a6bd563507c2661158c6a32d2e2b4dbc2276118f92e7f5082957d56da\": container with ID starting with e88bae5a6bd563507c2661158c6a32d2e2b4dbc2276118f92e7f5082957d56da not found: ID does not exist" containerID="e88bae5a6bd563507c2661158c6a32d2e2b4dbc2276118f92e7f5082957d56da" Jul 28 22:48:58.844685 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:58.844564 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e88bae5a6bd563507c2661158c6a32d2e2b4dbc2276118f92e7f5082957d56da"} err="failed to get container status \"e88bae5a6bd563507c2661158c6a32d2e2b4dbc2276118f92e7f5082957d56da\": rpc error: code = NotFound desc = could not find container \"e88bae5a6bd563507c2661158c6a32d2e2b4dbc2276118f92e7f5082957d56da\": container with ID starting with e88bae5a6bd563507c2661158c6a32d2e2b4dbc2276118f92e7f5082957d56da not found: ID does not exist" Jul 28 22:48:58.847434 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:58.847414 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j"] Jul 28 22:48:59.048838 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:59.048805 2553 patch_prober.go:28] interesting pod/e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 28 22:48:59.049022 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:48:59.048863 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-63f77a-openshift-default-55f9d74688-b9k9j" podUID="a31e333e-b17f-44e6-bedc-612fcf103228" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 28 22:49:00.413043 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:00.413012 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a31e333e-b17f-44e6-bedc-612fcf103228" path="/var/lib/kubelet/pods/a31e333e-b17f-44e6-bedc-612fcf103228/volumes" Jul 28 22:49:04.271531 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:04.271494 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-472fa3-5466868976-jv2fk"] Jul 28 22:49:04.272299 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:04.272269 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-472fa3-5466868976-jv2fk" podUID="625eb68e-7c1b-4758-8e74-2d2052645fff" containerName="maas-api" containerID="cri-o://729c41751b2163a0aef21b8812f89a8094ac0001105b52abe2a7cdf8fd5a4170" gracePeriod=30 Jul 28 22:49:04.525352 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:04.525285 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-472fa3-5466868976-jv2fk" Jul 28 22:49:04.694538 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:04.694496 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/625eb68e-7c1b-4758-8e74-2d2052645fff-maas-api-tls\") pod \"625eb68e-7c1b-4758-8e74-2d2052645fff\" (UID: \"625eb68e-7c1b-4758-8e74-2d2052645fff\") " Jul 28 22:49:04.694757 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:04.694587 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wfzhn\" (UniqueName: \"kubernetes.io/projected/625eb68e-7c1b-4758-8e74-2d2052645fff-kube-api-access-wfzhn\") pod \"625eb68e-7c1b-4758-8e74-2d2052645fff\" (UID: \"625eb68e-7c1b-4758-8e74-2d2052645fff\") " Jul 28 22:49:04.694757 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:04.694620 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/625eb68e-7c1b-4758-8e74-2d2052645fff-openshift-service-ca\") pod \"625eb68e-7c1b-4758-8e74-2d2052645fff\" (UID: \"625eb68e-7c1b-4758-8e74-2d2052645fff\") " Jul 28 22:49:04.695050 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:04.695019 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/625eb68e-7c1b-4758-8e74-2d2052645fff-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "625eb68e-7c1b-4758-8e74-2d2052645fff" (UID: "625eb68e-7c1b-4758-8e74-2d2052645fff"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 22:49:04.696917 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:04.696896 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/625eb68e-7c1b-4758-8e74-2d2052645fff-kube-api-access-wfzhn" (OuterVolumeSpecName: "kube-api-access-wfzhn") pod "625eb68e-7c1b-4758-8e74-2d2052645fff" (UID: "625eb68e-7c1b-4758-8e74-2d2052645fff"). InnerVolumeSpecName "kube-api-access-wfzhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:49:04.697026 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:04.696938 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/625eb68e-7c1b-4758-8e74-2d2052645fff-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "625eb68e-7c1b-4758-8e74-2d2052645fff" (UID: "625eb68e-7c1b-4758-8e74-2d2052645fff"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 22:49:04.796099 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:04.796003 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfzhn\" (UniqueName: \"kubernetes.io/projected/625eb68e-7c1b-4758-8e74-2d2052645fff-kube-api-access-wfzhn\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:49:04.796099 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:04.796040 2553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/625eb68e-7c1b-4758-8e74-2d2052645fff-openshift-service-ca\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:49:04.796099 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:04.796050 2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/625eb68e-7c1b-4758-8e74-2d2052645fff-maas-api-tls\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:49:04.858608 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:04.858572 2553 generic.go:356] "Generic (PLEG): container finished" podID="625eb68e-7c1b-4758-8e74-2d2052645fff" containerID="729c41751b2163a0aef21b8812f89a8094ac0001105b52abe2a7cdf8fd5a4170" exitCode=0 Jul 28 22:49:04.858816 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:04.858617 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-472fa3-5466868976-jv2fk" Jul 28 22:49:04.858816 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:04.858658 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-472fa3-5466868976-jv2fk" event={"ID":"625eb68e-7c1b-4758-8e74-2d2052645fff","Type":"ContainerDied","Data":"729c41751b2163a0aef21b8812f89a8094ac0001105b52abe2a7cdf8fd5a4170"} Jul 28 22:49:04.858816 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:04.858726 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-472fa3-5466868976-jv2fk" event={"ID":"625eb68e-7c1b-4758-8e74-2d2052645fff","Type":"ContainerDied","Data":"d521f1735c18b6662b7b7b133b5bf29286aae4f8b582a908a596d8dbb0c3614c"} Jul 28 22:49:04.858816 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:04.858753 2553 scope.go:117] "RemoveContainer" containerID="729c41751b2163a0aef21b8812f89a8094ac0001105b52abe2a7cdf8fd5a4170" Jul 28 22:49:04.867590 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:04.867572 2553 scope.go:117] "RemoveContainer" containerID="729c41751b2163a0aef21b8812f89a8094ac0001105b52abe2a7cdf8fd5a4170" Jul 28 22:49:04.867889 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:49:04.867870 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"729c41751b2163a0aef21b8812f89a8094ac0001105b52abe2a7cdf8fd5a4170\": container with ID starting with 729c41751b2163a0aef21b8812f89a8094ac0001105b52abe2a7cdf8fd5a4170 not found: ID does not exist" containerID="729c41751b2163a0aef21b8812f89a8094ac0001105b52abe2a7cdf8fd5a4170" Jul 28 22:49:04.867944 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:04.867897 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"729c41751b2163a0aef21b8812f89a8094ac0001105b52abe2a7cdf8fd5a4170"} err="failed to get container status \"729c41751b2163a0aef21b8812f89a8094ac0001105b52abe2a7cdf8fd5a4170\": rpc error: code = NotFound desc = could not find container \"729c41751b2163a0aef21b8812f89a8094ac0001105b52abe2a7cdf8fd5a4170\": container with ID starting with 729c41751b2163a0aef21b8812f89a8094ac0001105b52abe2a7cdf8fd5a4170 not found: ID does not exist" Jul 28 22:49:04.878591 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:04.878568 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-472fa3-5466868976-jv2fk"] Jul 28 22:49:04.884382 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:04.884361 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-472fa3-5466868976-jv2fk"] Jul 28 22:49:06.405421 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:06.405385 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="625eb68e-7c1b-4758-8e74-2d2052645fff" path="/var/lib/kubelet/pods/625eb68e-7c1b-4758-8e74-2d2052645fff/volumes" Jul 28 22:49:10.817205 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:10.817171 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9"] Jul 28 22:49:10.817680 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:10.817414 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9" podUID="11ea77c8-026d-45c0-8023-f2d4a965666b" containerName="istio-proxy" containerID="cri-o://3d05e7be2d0bc47104fee2651858d5f7784948c3cd2ce69fa17e52bd283e0295" gracePeriod=30 Jul 28 22:49:11.691258 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:11.691225 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh"] Jul 28 22:49:11.691773 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:11.691752 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a31e333e-b17f-44e6-bedc-612fcf103228" containerName="istio-proxy" Jul 28 22:49:11.691773 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:11.691774 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="a31e333e-b17f-44e6-bedc-612fcf103228" containerName="istio-proxy" Jul 28 22:49:11.691953 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:11.691796 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="625eb68e-7c1b-4758-8e74-2d2052645fff" containerName="maas-api" Jul 28 22:49:11.691953 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:11.691804 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="625eb68e-7c1b-4758-8e74-2d2052645fff" containerName="maas-api" Jul 28 22:49:11.691953 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:11.691912 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="625eb68e-7c1b-4758-8e74-2d2052645fff" containerName="maas-api" Jul 28 22:49:11.691953 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:11.691928 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="a31e333e-b17f-44e6-bedc-612fcf103228" containerName="istio-proxy" Jul 28 22:49:11.695036 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:11.695017 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh" Jul 28 22:49:11.697290 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:11.697266 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-a134df-openshift-default-dockercfg-q7g66\"" Jul 28 22:49:11.705512 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:11.705489 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh"] Jul 28 22:49:11.862304 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:11.862264 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04fb80a-7dab-4e27-84c3-687d81d5ce99-credential-socket\") pod \"e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh\" (UID: \"c04fb80a-7dab-4e27-84c3-687d81d5ce99\") " pod="openshift-ingress/e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh" Jul 28 22:49:11.862304 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:11.862308 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c04fb80a-7dab-4e27-84c3-687d81d5ce99-istio-token\") pod \"e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh\" (UID: \"c04fb80a-7dab-4e27-84c3-687d81d5ce99\") " pod="openshift-ingress/e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh" Jul 28 22:49:11.862829 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:11.862360 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04fb80a-7dab-4e27-84c3-687d81d5ce99-workload-socket\") pod \"e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh\" (UID: \"c04fb80a-7dab-4e27-84c3-687d81d5ce99\") " pod="openshift-ingress/e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh" Jul 28 22:49:11.862829 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:11.862458 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c04fb80a-7dab-4e27-84c3-687d81d5ce99-istio-data\") pod \"e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh\" (UID: \"c04fb80a-7dab-4e27-84c3-687d81d5ce99\") " pod="openshift-ingress/e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh" Jul 28 22:49:11.862829 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:11.862479 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c04fb80a-7dab-4e27-84c3-687d81d5ce99-istiod-ca-cert\") pod \"e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh\" (UID: \"c04fb80a-7dab-4e27-84c3-687d81d5ce99\") " pod="openshift-ingress/e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh" Jul 28 22:49:11.862829 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:11.862504 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c04fb80a-7dab-4e27-84c3-687d81d5ce99-istio-envoy\") pod \"e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh\" (UID: \"c04fb80a-7dab-4e27-84c3-687d81d5ce99\") " pod="openshift-ingress/e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh" Jul 28 22:49:11.862829 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:11.862534 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7f2v\" (UniqueName: \"kubernetes.io/projected/c04fb80a-7dab-4e27-84c3-687d81d5ce99-kube-api-access-w7f2v\") pod \"e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh\" (UID: \"c04fb80a-7dab-4e27-84c3-687d81d5ce99\") " pod="openshift-ingress/e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh" Jul 28 22:49:11.862829 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:11.862592 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c04fb80a-7dab-4e27-84c3-687d81d5ce99-istio-podinfo\") pod \"e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh\" (UID: \"c04fb80a-7dab-4e27-84c3-687d81d5ce99\") " pod="openshift-ingress/e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh" Jul 28 22:49:11.862829 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:11.862655 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c04fb80a-7dab-4e27-84c3-687d81d5ce99-workload-certs\") pod \"e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh\" (UID: \"c04fb80a-7dab-4e27-84c3-687d81d5ce99\") " pod="openshift-ingress/e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh" Jul 28 22:49:11.963713 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:11.963672 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c04fb80a-7dab-4e27-84c3-687d81d5ce99-istio-podinfo\") pod \"e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh\" (UID: \"c04fb80a-7dab-4e27-84c3-687d81d5ce99\") " pod="openshift-ingress/e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh" Jul 28 22:49:11.963893 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:11.963748 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c04fb80a-7dab-4e27-84c3-687d81d5ce99-workload-certs\") pod \"e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh\" (UID: \"c04fb80a-7dab-4e27-84c3-687d81d5ce99\") " pod="openshift-ingress/e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh" Jul 28 22:49:11.963893 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:11.963787 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04fb80a-7dab-4e27-84c3-687d81d5ce99-credential-socket\") pod \"e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh\" (UID: \"c04fb80a-7dab-4e27-84c3-687d81d5ce99\") " pod="openshift-ingress/e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh" Jul 28 22:49:11.963893 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:11.963809 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c04fb80a-7dab-4e27-84c3-687d81d5ce99-istio-token\") pod \"e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh\" (UID: \"c04fb80a-7dab-4e27-84c3-687d81d5ce99\") " pod="openshift-ingress/e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh" Jul 28 22:49:11.963893 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:11.963839 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04fb80a-7dab-4e27-84c3-687d81d5ce99-workload-socket\") pod \"e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh\" (UID: \"c04fb80a-7dab-4e27-84c3-687d81d5ce99\") " pod="openshift-ingress/e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh" Jul 28 22:49:11.963893 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:11.963865 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c04fb80a-7dab-4e27-84c3-687d81d5ce99-istio-data\") pod \"e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh\" (UID: \"c04fb80a-7dab-4e27-84c3-687d81d5ce99\") " pod="openshift-ingress/e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh" Jul 28 22:49:11.963893 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:11.963889 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c04fb80a-7dab-4e27-84c3-687d81d5ce99-istiod-ca-cert\") pod \"e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh\" (UID: \"c04fb80a-7dab-4e27-84c3-687d81d5ce99\") " pod="openshift-ingress/e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh" Jul 28 22:49:11.964209 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:11.963927 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c04fb80a-7dab-4e27-84c3-687d81d5ce99-istio-envoy\") pod \"e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh\" (UID: \"c04fb80a-7dab-4e27-84c3-687d81d5ce99\") " pod="openshift-ingress/e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh" Jul 28 22:49:11.964209 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:11.963969 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7f2v\" (UniqueName: \"kubernetes.io/projected/c04fb80a-7dab-4e27-84c3-687d81d5ce99-kube-api-access-w7f2v\") pod \"e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh\" (UID: \"c04fb80a-7dab-4e27-84c3-687d81d5ce99\") " pod="openshift-ingress/e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh" Jul 28 22:49:11.964314 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:11.964241 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c04fb80a-7dab-4e27-84c3-687d81d5ce99-workload-certs\") pod \"e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh\" (UID: \"c04fb80a-7dab-4e27-84c3-687d81d5ce99\") " pod="openshift-ingress/e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh" Jul 28 22:49:11.964314 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:11.964274 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04fb80a-7dab-4e27-84c3-687d81d5ce99-workload-socket\") pod \"e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh\" (UID: \"c04fb80a-7dab-4e27-84c3-687d81d5ce99\") " pod="openshift-ingress/e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh" Jul 28 22:49:11.964482 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:11.964465 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c04fb80a-7dab-4e27-84c3-687d81d5ce99-istio-data\") pod \"e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh\" (UID: \"c04fb80a-7dab-4e27-84c3-687d81d5ce99\") " pod="openshift-ingress/e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh" Jul 28 22:49:11.964554 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:11.964503 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04fb80a-7dab-4e27-84c3-687d81d5ce99-credential-socket\") pod \"e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh\" (UID: \"c04fb80a-7dab-4e27-84c3-687d81d5ce99\") " pod="openshift-ingress/e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh" Jul 28 22:49:11.964818 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:11.964799 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c04fb80a-7dab-4e27-84c3-687d81d5ce99-istiod-ca-cert\") pod \"e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh\" (UID: \"c04fb80a-7dab-4e27-84c3-687d81d5ce99\") " pod="openshift-ingress/e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh" Jul 28 22:49:11.966255 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:11.966235 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c04fb80a-7dab-4e27-84c3-687d81d5ce99-istio-podinfo\") pod \"e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh\" (UID: \"c04fb80a-7dab-4e27-84c3-687d81d5ce99\") " pod="openshift-ingress/e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh" Jul 28 22:49:11.966364 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:11.966346 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c04fb80a-7dab-4e27-84c3-687d81d5ce99-istio-envoy\") pod \"e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh\" (UID: \"c04fb80a-7dab-4e27-84c3-687d81d5ce99\") " pod="openshift-ingress/e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh" Jul 28 22:49:11.971399 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:11.971375 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c04fb80a-7dab-4e27-84c3-687d81d5ce99-istio-token\") pod \"e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh\" (UID: \"c04fb80a-7dab-4e27-84c3-687d81d5ce99\") " pod="openshift-ingress/e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh" Jul 28 22:49:11.971578 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:11.971561 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7f2v\" (UniqueName: \"kubernetes.io/projected/c04fb80a-7dab-4e27-84c3-687d81d5ce99-kube-api-access-w7f2v\") pod \"e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh\" (UID: \"c04fb80a-7dab-4e27-84c3-687d81d5ce99\") " pod="openshift-ingress/e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh" Jul 28 22:49:12.006163 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:12.006126 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh" Jul 28 22:49:12.152480 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:12.152453 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh"] Jul 28 22:49:12.154410 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:49:12.154387 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc04fb80a_7dab_4e27_84c3_687d81d5ce99.slice/crio-7799a9fb7010a9d9540534fd53470adeaa03779bc4174f7ec8a8a7ebe80f0e2c WatchSource:0}: Error finding container 7799a9fb7010a9d9540534fd53470adeaa03779bc4174f7ec8a8a7ebe80f0e2c: Status 404 returned error can't find the container with id 7799a9fb7010a9d9540534fd53470adeaa03779bc4174f7ec8a8a7ebe80f0e2c Jul 28 22:49:12.156547 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:12.156512 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:49:12.156736 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:12.156593 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:49:12.156736 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:12.156637 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:49:12.905596 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:12.905555 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh" event={"ID":"c04fb80a-7dab-4e27-84c3-687d81d5ce99","Type":"ContainerStarted","Data":"848dca6aef0c8293024aa657d213ec40f085bde8d09fd00f6b430cf521bbf347"} Jul 28 22:49:12.906250 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:12.905620 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh" event={"ID":"c04fb80a-7dab-4e27-84c3-687d81d5ce99","Type":"ContainerStarted","Data":"7799a9fb7010a9d9540534fd53470adeaa03779bc4174f7ec8a8a7ebe80f0e2c"} Jul 28 22:49:12.927005 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:12.926949 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh" podStartSLOduration=1.926934112 podStartE2EDuration="1.926934112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:49:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:49:12.924153772 +0000 UTC m=+2373.011411841" watchObservedRunningTime="2026-07-28 22:49:12.926934112 +0000 UTC m=+2373.014192180" Jul 28 22:49:13.007240 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:13.007208 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh" Jul 28 22:49:13.012098 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:13.012076 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh" Jul 28 22:49:13.362344 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:13.362309 2553 patch_prober.go:28] interesting pod/e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 28 22:49:13.362497 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:13.362365 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9" podUID="11ea77c8-026d-45c0-8023-f2d4a965666b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 22:49:13.911215 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:13.911174 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh" Jul 28 22:49:13.912065 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:13.912043 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh" Jul 28 22:49:14.891173 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:14.891142 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a134df-5485979d88-p2msq"] Jul 28 22:49:14.891593 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:14.891573 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a31e333e-b17f-44e6-bedc-612fcf103228" containerName="istio-proxy" Jul 28 22:49:14.891694 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:14.891596 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="a31e333e-b17f-44e6-bedc-612fcf103228" containerName="istio-proxy" Jul 28 22:49:14.891694 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:14.891689 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="a31e333e-b17f-44e6-bedc-612fcf103228" containerName="istio-proxy" Jul 28 22:49:14.894892 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:14.894871 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a134df-5485979d88-p2msq" Jul 28 22:49:14.897085 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:14.897063 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-a134df-serving-cert\"" Jul 28 22:49:14.907100 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:14.907066 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a134df-5485979d88-p2msq"] Jul 28 22:49:14.992491 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:14.992451 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72cb9aea-6e37-40c7-9a68-af24a8226469-openshift-service-ca\") pod \"maas-api-e2e-shared-a-a134df-5485979d88-p2msq\" (UID: \"72cb9aea-6e37-40c7-9a68-af24a8226469\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a134df-5485979d88-p2msq" Jul 28 22:49:14.992954 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:14.992625 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72cb9aea-6e37-40c7-9a68-af24a8226469-maas-api-tls\") pod \"maas-api-e2e-shared-a-a134df-5485979d88-p2msq\" (UID: \"72cb9aea-6e37-40c7-9a68-af24a8226469\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a134df-5485979d88-p2msq" Jul 28 22:49:14.992954 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:14.992755 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-594h5\" (UniqueName: \"kubernetes.io/projected/72cb9aea-6e37-40c7-9a68-af24a8226469-kube-api-access-594h5\") pod \"maas-api-e2e-shared-a-a134df-5485979d88-p2msq\" (UID: \"72cb9aea-6e37-40c7-9a68-af24a8226469\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a134df-5485979d88-p2msq" Jul 28 22:49:15.093392 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:15.093354 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72cb9aea-6e37-40c7-9a68-af24a8226469-openshift-service-ca\") pod \"maas-api-e2e-shared-a-a134df-5485979d88-p2msq\" (UID: \"72cb9aea-6e37-40c7-9a68-af24a8226469\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a134df-5485979d88-p2msq" Jul 28 22:49:15.093575 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:15.093407 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72cb9aea-6e37-40c7-9a68-af24a8226469-maas-api-tls\") pod \"maas-api-e2e-shared-a-a134df-5485979d88-p2msq\" (UID: \"72cb9aea-6e37-40c7-9a68-af24a8226469\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a134df-5485979d88-p2msq" Jul 28 22:49:15.093575 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:15.093431 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-594h5\" (UniqueName: \"kubernetes.io/projected/72cb9aea-6e37-40c7-9a68-af24a8226469-kube-api-access-594h5\") pod \"maas-api-e2e-shared-a-a134df-5485979d88-p2msq\" (UID: \"72cb9aea-6e37-40c7-9a68-af24a8226469\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a134df-5485979d88-p2msq" Jul 28 22:49:15.093575 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:49:15.093561 2553 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-a134df-serving-cert: secret "maas-api-e2e-shared-a-a134df-serving-cert" not found Jul 28 22:49:15.093740 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:49:15.093625 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72cb9aea-6e37-40c7-9a68-af24a8226469-maas-api-tls podName:72cb9aea-6e37-40c7-9a68-af24a8226469 nodeName:}" failed. No retries permitted until 2026-07-28 22:49:15.593609767 +0000 UTC m=+2375.680867818 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/72cb9aea-6e37-40c7-9a68-af24a8226469-maas-api-tls") pod "maas-api-e2e-shared-a-a134df-5485979d88-p2msq" (UID: "72cb9aea-6e37-40c7-9a68-af24a8226469") : secret "maas-api-e2e-shared-a-a134df-serving-cert" not found Jul 28 22:49:15.094027 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:15.094009 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72cb9aea-6e37-40c7-9a68-af24a8226469-openshift-service-ca\") pod \"maas-api-e2e-shared-a-a134df-5485979d88-p2msq\" (UID: \"72cb9aea-6e37-40c7-9a68-af24a8226469\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a134df-5485979d88-p2msq" Jul 28 22:49:15.106684 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:15.106649 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-594h5\" (UniqueName: \"kubernetes.io/projected/72cb9aea-6e37-40c7-9a68-af24a8226469-kube-api-access-594h5\") pod \"maas-api-e2e-shared-a-a134df-5485979d88-p2msq\" (UID: \"72cb9aea-6e37-40c7-9a68-af24a8226469\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a134df-5485979d88-p2msq" Jul 28 22:49:15.595845 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:15.595809 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72cb9aea-6e37-40c7-9a68-af24a8226469-maas-api-tls\") pod \"maas-api-e2e-shared-a-a134df-5485979d88-p2msq\" (UID: \"72cb9aea-6e37-40c7-9a68-af24a8226469\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a134df-5485979d88-p2msq" Jul 28 22:49:15.596028 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:49:15.595951 2553 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-a134df-serving-cert: secret "maas-api-e2e-shared-a-a134df-serving-cert" not found Jul 28 22:49:15.596028 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:49:15.596022 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72cb9aea-6e37-40c7-9a68-af24a8226469-maas-api-tls podName:72cb9aea-6e37-40c7-9a68-af24a8226469 nodeName:}" failed. No retries permitted until 2026-07-28 22:49:16.596006967 +0000 UTC m=+2376.683265014 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/72cb9aea-6e37-40c7-9a68-af24a8226469-maas-api-tls") pod "maas-api-e2e-shared-a-a134df-5485979d88-p2msq" (UID: "72cb9aea-6e37-40c7-9a68-af24a8226469") : secret "maas-api-e2e-shared-a-a134df-serving-cert" not found Jul 28 22:49:16.069404 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:16.069372 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9" Jul 28 22:49:16.099256 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:16.099229 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ea77c8-026d-45c0-8023-f2d4a965666b-credential-socket\") pod \"11ea77c8-026d-45c0-8023-f2d4a965666b\" (UID: \"11ea77c8-026d-45c0-8023-f2d4a965666b\") " Jul 28 22:49:16.099436 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:16.099293 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11ea77c8-026d-45c0-8023-f2d4a965666b-istio-podinfo\") pod \"11ea77c8-026d-45c0-8023-f2d4a965666b\" (UID: \"11ea77c8-026d-45c0-8023-f2d4a965666b\") " Jul 28 22:49:16.099436 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:16.099317 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ea77c8-026d-45c0-8023-f2d4a965666b-workload-socket\") pod \"11ea77c8-026d-45c0-8023-f2d4a965666b\" (UID: \"11ea77c8-026d-45c0-8023-f2d4a965666b\") " Jul 28 22:49:16.099436 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:16.099356 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11ea77c8-026d-45c0-8023-f2d4a965666b-istiod-ca-cert\") pod \"11ea77c8-026d-45c0-8023-f2d4a965666b\" (UID: \"11ea77c8-026d-45c0-8023-f2d4a965666b\") " Jul 28 22:49:16.099436 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:16.099405 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11ea77c8-026d-45c0-8023-f2d4a965666b-istio-token\") pod \"11ea77c8-026d-45c0-8023-f2d4a965666b\" (UID: \"11ea77c8-026d-45c0-8023-f2d4a965666b\") " Jul 28 22:49:16.099652 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:16.099454 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8hffq\" (UniqueName: \"kubernetes.io/projected/11ea77c8-026d-45c0-8023-f2d4a965666b-kube-api-access-8hffq\") pod \"11ea77c8-026d-45c0-8023-f2d4a965666b\" (UID: \"11ea77c8-026d-45c0-8023-f2d4a965666b\") " Jul 28 22:49:16.099652 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:16.099484 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11ea77c8-026d-45c0-8023-f2d4a965666b-istio-data\") pod \"11ea77c8-026d-45c0-8023-f2d4a965666b\" (UID: \"11ea77c8-026d-45c0-8023-f2d4a965666b\") " Jul 28 22:49:16.099652 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:16.099490 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11ea77c8-026d-45c0-8023-f2d4a965666b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "11ea77c8-026d-45c0-8023-f2d4a965666b" (UID: "11ea77c8-026d-45c0-8023-f2d4a965666b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:49:16.099652 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:16.099517 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11ea77c8-026d-45c0-8023-f2d4a965666b-workload-certs\") pod \"11ea77c8-026d-45c0-8023-f2d4a965666b\" (UID: \"11ea77c8-026d-45c0-8023-f2d4a965666b\") " Jul 28 22:49:16.099652 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:16.099584 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11ea77c8-026d-45c0-8023-f2d4a965666b-istio-envoy\") pod \"11ea77c8-026d-45c0-8023-f2d4a965666b\" (UID: \"11ea77c8-026d-45c0-8023-f2d4a965666b\") " Jul 28 22:49:16.099927 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:16.099747 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11ea77c8-026d-45c0-8023-f2d4a965666b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "11ea77c8-026d-45c0-8023-f2d4a965666b" (UID: "11ea77c8-026d-45c0-8023-f2d4a965666b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 22:49:16.099990 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:16.099865 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11ea77c8-026d-45c0-8023-f2d4a965666b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "11ea77c8-026d-45c0-8023-f2d4a965666b" (UID: "11ea77c8-026d-45c0-8023-f2d4a965666b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:49:16.100054 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:16.099981 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11ea77c8-026d-45c0-8023-f2d4a965666b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "11ea77c8-026d-45c0-8023-f2d4a965666b" (UID: "11ea77c8-026d-45c0-8023-f2d4a965666b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:49:16.100054 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:16.100034 2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ea77c8-026d-45c0-8023-f2d4a965666b-credential-socket\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:49:16.100162 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:16.100056 2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11ea77c8-026d-45c0-8023-f2d4a965666b-istiod-ca-cert\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:49:16.100993 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:16.100967 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11ea77c8-026d-45c0-8023-f2d4a965666b-istio-data" (OuterVolumeSpecName: "istio-data") pod "11ea77c8-026d-45c0-8023-f2d4a965666b" (UID: "11ea77c8-026d-45c0-8023-f2d4a965666b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:49:16.101972 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:16.101939 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11ea77c8-026d-45c0-8023-f2d4a965666b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "11ea77c8-026d-45c0-8023-f2d4a965666b" (UID: "11ea77c8-026d-45c0-8023-f2d4a965666b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 22:49:16.102074 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:16.102000 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11ea77c8-026d-45c0-8023-f2d4a965666b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "11ea77c8-026d-45c0-8023-f2d4a965666b" (UID: "11ea77c8-026d-45c0-8023-f2d4a965666b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 22:49:16.102206 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:16.102186 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11ea77c8-026d-45c0-8023-f2d4a965666b-kube-api-access-8hffq" (OuterVolumeSpecName: "kube-api-access-8hffq") pod "11ea77c8-026d-45c0-8023-f2d4a965666b" (UID: "11ea77c8-026d-45c0-8023-f2d4a965666b"). InnerVolumeSpecName "kube-api-access-8hffq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:49:16.102808 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:16.102766 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11ea77c8-026d-45c0-8023-f2d4a965666b-istio-token" (OuterVolumeSpecName: "istio-token") pod "11ea77c8-026d-45c0-8023-f2d4a965666b" (UID: "11ea77c8-026d-45c0-8023-f2d4a965666b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 22:49:16.200528 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:16.200450 2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11ea77c8-026d-45c0-8023-f2d4a965666b-istio-data\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:49:16.200528 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:16.200478 2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11ea77c8-026d-45c0-8023-f2d4a965666b-workload-certs\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:49:16.200528 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:16.200488 2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11ea77c8-026d-45c0-8023-f2d4a965666b-istio-envoy\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:49:16.200528 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:16.200496 2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11ea77c8-026d-45c0-8023-f2d4a965666b-istio-podinfo\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:49:16.200528 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:16.200503 2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ea77c8-026d-45c0-8023-f2d4a965666b-workload-socket\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:49:16.200528 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:16.200514 2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11ea77c8-026d-45c0-8023-f2d4a965666b-istio-token\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:49:16.200528 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:16.200524 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hffq\" (UniqueName: \"kubernetes.io/projected/11ea77c8-026d-45c0-8023-f2d4a965666b-kube-api-access-8hffq\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 22:49:16.608603 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:16.608559 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72cb9aea-6e37-40c7-9a68-af24a8226469-maas-api-tls\") pod \"maas-api-e2e-shared-a-a134df-5485979d88-p2msq\" (UID: \"72cb9aea-6e37-40c7-9a68-af24a8226469\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a134df-5485979d88-p2msq" Jul 28 22:49:16.611385 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:16.611351 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72cb9aea-6e37-40c7-9a68-af24a8226469-maas-api-tls\") pod \"maas-api-e2e-shared-a-a134df-5485979d88-p2msq\" (UID: \"72cb9aea-6e37-40c7-9a68-af24a8226469\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a134df-5485979d88-p2msq" Jul 28 22:49:16.707531 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:16.707497 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a134df-5485979d88-p2msq" Jul 28 22:49:16.866997 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:16.866969 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a134df-5485979d88-p2msq"] Jul 28 22:49:16.869103 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:49:16.869075 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72cb9aea_6e37_40c7_9a68_af24a8226469.slice/crio-43271b505298ea4005de13d4e8f4a46ab597b93b261471f2600dec00b5347e88 WatchSource:0}: Error finding container 43271b505298ea4005de13d4e8f4a46ab597b93b261471f2600dec00b5347e88: Status 404 returned error can't find the container with id 43271b505298ea4005de13d4e8f4a46ab597b93b261471f2600dec00b5347e88 Jul 28 22:49:16.930848 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:16.930811 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a134df-5485979d88-p2msq" event={"ID":"72cb9aea-6e37-40c7-9a68-af24a8226469","Type":"ContainerStarted","Data":"43271b505298ea4005de13d4e8f4a46ab597b93b261471f2600dec00b5347e88"} Jul 28 22:49:16.932411 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:16.932388 2553 generic.go:356] "Generic (PLEG): container finished" podID="11ea77c8-026d-45c0-8023-f2d4a965666b" containerID="3d05e7be2d0bc47104fee2651858d5f7784948c3cd2ce69fa17e52bd283e0295" exitCode=0 Jul 28 22:49:16.932529 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:16.932424 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9" event={"ID":"11ea77c8-026d-45c0-8023-f2d4a965666b","Type":"ContainerDied","Data":"3d05e7be2d0bc47104fee2651858d5f7784948c3cd2ce69fa17e52bd283e0295"} Jul 28 22:49:16.932529 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:16.932447 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9" Jul 28 22:49:16.932529 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:16.932462 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9" event={"ID":"11ea77c8-026d-45c0-8023-f2d4a965666b","Type":"ContainerDied","Data":"cb81ffcc9ff136f9667c1ea0d351fa6b2be86718202453692baef1a0d2a12ee8"} Jul 28 22:49:16.932529 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:16.932482 2553 scope.go:117] "RemoveContainer" containerID="3d05e7be2d0bc47104fee2651858d5f7784948c3cd2ce69fa17e52bd283e0295" Jul 28 22:49:16.942109 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:16.941936 2553 scope.go:117] "RemoveContainer" containerID="39431c32e51f72ba04a7657c3e9fcb49746a1c1fe138779f8ca46db96d3a9564" Jul 28 22:49:16.951210 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:16.951175 2553 scope.go:117] "RemoveContainer" containerID="3d05e7be2d0bc47104fee2651858d5f7784948c3cd2ce69fa17e52bd283e0295" Jul 28 22:49:16.951665 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:49:16.951641 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d05e7be2d0bc47104fee2651858d5f7784948c3cd2ce69fa17e52bd283e0295\": container with ID starting with 3d05e7be2d0bc47104fee2651858d5f7784948c3cd2ce69fa17e52bd283e0295 not found: ID does not exist" containerID="3d05e7be2d0bc47104fee2651858d5f7784948c3cd2ce69fa17e52bd283e0295" Jul 28 22:49:16.951882 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:16.951675 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d05e7be2d0bc47104fee2651858d5f7784948c3cd2ce69fa17e52bd283e0295"} err="failed to get container status \"3d05e7be2d0bc47104fee2651858d5f7784948c3cd2ce69fa17e52bd283e0295\": rpc error: code = NotFound desc = could not find container \"3d05e7be2d0bc47104fee2651858d5f7784948c3cd2ce69fa17e52bd283e0295\": container with ID starting with 3d05e7be2d0bc47104fee2651858d5f7784948c3cd2ce69fa17e52bd283e0295 not found: ID does not exist" Jul 28 22:49:16.951882 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:16.951694 2553 scope.go:117] "RemoveContainer" containerID="39431c32e51f72ba04a7657c3e9fcb49746a1c1fe138779f8ca46db96d3a9564" Jul 28 22:49:16.952054 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:49:16.952031 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39431c32e51f72ba04a7657c3e9fcb49746a1c1fe138779f8ca46db96d3a9564\": container with ID starting with 39431c32e51f72ba04a7657c3e9fcb49746a1c1fe138779f8ca46db96d3a9564 not found: ID does not exist" containerID="39431c32e51f72ba04a7657c3e9fcb49746a1c1fe138779f8ca46db96d3a9564" Jul 28 22:49:16.952100 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:16.952066 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39431c32e51f72ba04a7657c3e9fcb49746a1c1fe138779f8ca46db96d3a9564"} err="failed to get container status \"39431c32e51f72ba04a7657c3e9fcb49746a1c1fe138779f8ca46db96d3a9564\": rpc error: code = NotFound desc = could not find container \"39431c32e51f72ba04a7657c3e9fcb49746a1c1fe138779f8ca46db96d3a9564\": container with ID starting with 39431c32e51f72ba04a7657c3e9fcb49746a1c1fe138779f8ca46db96d3a9564 not found: ID does not exist" Jul 28 22:49:16.953601 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:16.953581 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9"] Jul 28 22:49:16.955695 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:16.955677 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-472fa3-openshift-default-659b85747b-zmkt9"] Jul 28 22:49:17.939344 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:17.939307 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a134df-5485979d88-p2msq" event={"ID":"72cb9aea-6e37-40c7-9a68-af24a8226469","Type":"ContainerStarted","Data":"ba650cd157ebc7eafa38ed69fde28aafc2acbb72d929fad21a05f1825d1e146a"} Jul 28 22:49:17.939845 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:17.939373 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a134df-5485979d88-p2msq" Jul 28 22:49:17.954644 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:17.954593 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a134df-5485979d88-p2msq" podStartSLOduration=3.603046356 podStartE2EDuration="3.954577188s" totalImagesPullingTime="351.530832ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:49:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:49:17.952813617 +0000 UTC m=+2378.040071690" watchObservedRunningTime="2026-07-28 22:49:17.954577188 +0000 UTC m=+2378.041835284" Jul 28 22:49:18.404881 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:18.404847 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11ea77c8-026d-45c0-8023-f2d4a965666b" path="/var/lib/kubelet/pods/11ea77c8-026d-45c0-8023-f2d4a965666b/volumes" Jul 28 22:49:23.951297 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:23.951261 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a134df-5485979d88-p2msq" Jul 28 22:49:28.000785 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:28.000745 2553 generic.go:356] "Generic (PLEG): container finished" podID="c04fb80a-7dab-4e27-84c3-687d81d5ce99" containerID="848dca6aef0c8293024aa657d213ec40f085bde8d09fd00f6b430cf521bbf347" exitCode=0 Jul 28 22:49:28.001228 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:28.000818 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh" event={"ID":"c04fb80a-7dab-4e27-84c3-687d81d5ce99","Type":"ContainerDied","Data":"848dca6aef0c8293024aa657d213ec40f085bde8d09fd00f6b430cf521bbf347"} Jul 28 22:49:28.001279 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:28.001241 2553 scope.go:117] "RemoveContainer" containerID="848dca6aef0c8293024aa657d213ec40f085bde8d09fd00f6b430cf521bbf347" Jul 28 22:49:28.002286 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:28.002258 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:49:28.002382 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:28.002326 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:49:28.002382 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:28.002357 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:49:28.911502 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:28.911470 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh" Jul 28 22:49:29.007831 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:29.007792 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh" event={"ID":"c04fb80a-7dab-4e27-84c3-687d81d5ce99","Type":"ContainerStarted","Data":"c5b6ebb9d07e9fa861690f1b7e6e777b7266f3730f5afb1d95c4850c1c54350f"} Jul 28 22:49:30.007214 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:30.007173 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh" Jul 28 22:49:30.008518 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:30.008490 2553 patch_prober.go:28] interesting pod/e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Jul 28 22:49:30.008866 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:30.008557 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh" podUID="c04fb80a-7dab-4e27-84c3-687d81d5ce99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Jul 28 22:49:31.006841 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:31.006804 2553 patch_prober.go:28] interesting pod/e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Jul 28 22:49:31.007038 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:31.006858 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh" podUID="c04fb80a-7dab-4e27-84c3-687d81d5ce99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Jul 28 22:49:32.006588 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:32.006550 2553 patch_prober.go:28] interesting pod/e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Jul 28 22:49:32.007034 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:32.006621 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh" podUID="c04fb80a-7dab-4e27-84c3-687d81d5ce99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Jul 28 22:49:33.006945 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:33.006914 2553 patch_prober.go:28] interesting pod/e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Jul 28 22:49:33.007330 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:33.006970 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh" podUID="c04fb80a-7dab-4e27-84c3-687d81d5ce99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Jul 28 22:49:34.011263 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:34.011235 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh" Jul 28 22:49:34.011687 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:34.011518 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh" Jul 28 22:49:34.012184 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:34.012162 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh" Jul 28 22:49:40.443950 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:40.443917 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4"] Jul 28 22:49:40.451086 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:40.445165 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11ea77c8-026d-45c0-8023-f2d4a965666b" containerName="istio-proxy" Jul 28 22:49:40.451086 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:40.445194 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="11ea77c8-026d-45c0-8023-f2d4a965666b" containerName="istio-proxy" Jul 28 22:49:40.451086 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:40.445224 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11ea77c8-026d-45c0-8023-f2d4a965666b" containerName="istio-proxy" Jul 28 22:49:40.451086 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:40.445233 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="11ea77c8-026d-45c0-8023-f2d4a965666b" containerName="istio-proxy" Jul 28 22:49:40.451086 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:40.445435 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="11ea77c8-026d-45c0-8023-f2d4a965666b" containerName="istio-proxy" Jul 28 22:49:40.451086 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:40.445454 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="11ea77c8-026d-45c0-8023-f2d4a965666b" containerName="istio-proxy" Jul 28 22:49:40.454767 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:40.454743 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4" Jul 28 22:49:40.457070 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:40.456994 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-faf483-openshift-default-dockercfg-slhhd\"" Jul 28 22:49:40.460688 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:40.460660 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4"] Jul 28 22:49:40.527176 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:40.527146 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mldhp_e1ced996-f136-4d3e-9c85-a5b7809b4517/console-operator/1.log" Jul 28 22:49:40.533491 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:40.533463 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdd68e6b-b347-4c96-b476-f3341089ff50-credential-socket\") pod \"e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4\" (UID: \"cdd68e6b-b347-4c96-b476-f3341089ff50\") " pod="openshift-ingress/e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4" Jul 28 22:49:40.533646 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:40.533553 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdd68e6b-b347-4c96-b476-f3341089ff50-istio-token\") pod \"e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4\" (UID: \"cdd68e6b-b347-4c96-b476-f3341089ff50\") " pod="openshift-ingress/e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4" Jul 28 22:49:40.533646 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:40.533580 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdd68e6b-b347-4c96-b476-f3341089ff50-istiod-ca-cert\") pod \"e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4\" (UID: \"cdd68e6b-b347-4c96-b476-f3341089ff50\") " pod="openshift-ingress/e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4" Jul 28 22:49:40.533787 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:40.533655 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdd68e6b-b347-4c96-b476-f3341089ff50-istio-data\") pod \"e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4\" (UID: \"cdd68e6b-b347-4c96-b476-f3341089ff50\") " pod="openshift-ingress/e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4" Jul 28 22:49:40.533787 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:40.533724 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdd68e6b-b347-4c96-b476-f3341089ff50-istio-envoy\") pod \"e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4\" (UID: \"cdd68e6b-b347-4c96-b476-f3341089ff50\") " pod="openshift-ingress/e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4" Jul 28 22:49:40.533787 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:40.533771 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mldhp_e1ced996-f136-4d3e-9c85-a5b7809b4517/console-operator/1.log" Jul 28 22:49:40.533934 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:40.533795 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t59g6\" (UniqueName: \"kubernetes.io/projected/cdd68e6b-b347-4c96-b476-f3341089ff50-kube-api-access-t59g6\") pod \"e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4\" (UID: \"cdd68e6b-b347-4c96-b476-f3341089ff50\") " pod="openshift-ingress/e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4" Jul 28 22:49:40.533934 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:40.533826 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdd68e6b-b347-4c96-b476-f3341089ff50-istio-podinfo\") pod \"e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4\" (UID: \"cdd68e6b-b347-4c96-b476-f3341089ff50\") " pod="openshift-ingress/e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4" Jul 28 22:49:40.533934 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:40.533857 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdd68e6b-b347-4c96-b476-f3341089ff50-workload-socket\") pod \"e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4\" (UID: \"cdd68e6b-b347-4c96-b476-f3341089ff50\") " pod="openshift-ingress/e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4" Jul 28 22:49:40.533934 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:40.533914 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdd68e6b-b347-4c96-b476-f3341089ff50-workload-certs\") pod \"e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4\" (UID: \"cdd68e6b-b347-4c96-b476-f3341089ff50\") " pod="openshift-ingress/e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4" Jul 28 22:49:40.634578 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:40.634541 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t59g6\" (UniqueName: \"kubernetes.io/projected/cdd68e6b-b347-4c96-b476-f3341089ff50-kube-api-access-t59g6\") pod \"e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4\" (UID: \"cdd68e6b-b347-4c96-b476-f3341089ff50\") " pod="openshift-ingress/e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4" Jul 28 22:49:40.634578 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:40.634575 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdd68e6b-b347-4c96-b476-f3341089ff50-istio-podinfo\") pod \"e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4\" (UID: \"cdd68e6b-b347-4c96-b476-f3341089ff50\") " pod="openshift-ingress/e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4" Jul 28 22:49:40.634881 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:40.634596 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdd68e6b-b347-4c96-b476-f3341089ff50-workload-socket\") pod \"e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4\" (UID: \"cdd68e6b-b347-4c96-b476-f3341089ff50\") " pod="openshift-ingress/e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4" Jul 28 22:49:40.634881 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:40.634612 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdd68e6b-b347-4c96-b476-f3341089ff50-workload-certs\") pod \"e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4\" (UID: \"cdd68e6b-b347-4c96-b476-f3341089ff50\") " pod="openshift-ingress/e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4" Jul 28 22:49:40.634881 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:40.634770 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdd68e6b-b347-4c96-b476-f3341089ff50-credential-socket\") pod \"e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4\" (UID: \"cdd68e6b-b347-4c96-b476-f3341089ff50\") " pod="openshift-ingress/e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4" Jul 28 22:49:40.634881 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:40.634825 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdd68e6b-b347-4c96-b476-f3341089ff50-istio-token\") pod \"e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4\" (UID: \"cdd68e6b-b347-4c96-b476-f3341089ff50\") " pod="openshift-ingress/e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4" Jul 28 22:49:40.634881 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:40.634852 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdd68e6b-b347-4c96-b476-f3341089ff50-istiod-ca-cert\") pod \"e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4\" (UID: \"cdd68e6b-b347-4c96-b476-f3341089ff50\") " pod="openshift-ingress/e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4" Jul 28 22:49:40.635141 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:40.634925 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdd68e6b-b347-4c96-b476-f3341089ff50-istio-data\") pod \"e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4\" (UID: \"cdd68e6b-b347-4c96-b476-f3341089ff50\") " pod="openshift-ingress/e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4" Jul 28 22:49:40.635141 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:40.634975 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdd68e6b-b347-4c96-b476-f3341089ff50-istio-envoy\") pod \"e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4\" (UID: \"cdd68e6b-b347-4c96-b476-f3341089ff50\") " pod="openshift-ingress/e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4" Jul 28 22:49:40.635251 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:40.635231 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdd68e6b-b347-4c96-b476-f3341089ff50-workload-socket\") pod \"e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4\" (UID: \"cdd68e6b-b347-4c96-b476-f3341089ff50\") " pod="openshift-ingress/e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4" Jul 28 22:49:40.635309 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:40.635287 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdd68e6b-b347-4c96-b476-f3341089ff50-credential-socket\") pod \"e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4\" (UID: \"cdd68e6b-b347-4c96-b476-f3341089ff50\") " pod="openshift-ingress/e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4" Jul 28 22:49:40.635418 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:40.635390 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdd68e6b-b347-4c96-b476-f3341089ff50-istio-data\") pod \"e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4\" (UID: \"cdd68e6b-b347-4c96-b476-f3341089ff50\") " pod="openshift-ingress/e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4" Jul 28 22:49:40.635618 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:40.635599 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdd68e6b-b347-4c96-b476-f3341089ff50-workload-certs\") pod \"e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4\" (UID: \"cdd68e6b-b347-4c96-b476-f3341089ff50\") " pod="openshift-ingress/e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4" Jul 28 22:49:40.635618 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:40.635610 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdd68e6b-b347-4c96-b476-f3341089ff50-istiod-ca-cert\") pod \"e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4\" (UID: \"cdd68e6b-b347-4c96-b476-f3341089ff50\") " pod="openshift-ingress/e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4" Jul 28 22:49:40.637330 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:40.637302 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdd68e6b-b347-4c96-b476-f3341089ff50-istio-podinfo\") pod \"e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4\" (UID: \"cdd68e6b-b347-4c96-b476-f3341089ff50\") " pod="openshift-ingress/e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4" Jul 28 22:49:40.637429 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:40.637361 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdd68e6b-b347-4c96-b476-f3341089ff50-istio-envoy\") pod \"e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4\" (UID: \"cdd68e6b-b347-4c96-b476-f3341089ff50\") " pod="openshift-ingress/e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4" Jul 28 22:49:40.642267 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:40.642242 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdd68e6b-b347-4c96-b476-f3341089ff50-istio-token\") pod \"e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4\" (UID: \"cdd68e6b-b347-4c96-b476-f3341089ff50\") " pod="openshift-ingress/e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4" Jul 28 22:49:40.642442 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:40.642422 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t59g6\" (UniqueName: \"kubernetes.io/projected/cdd68e6b-b347-4c96-b476-f3341089ff50-kube-api-access-t59g6\") pod \"e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4\" (UID: \"cdd68e6b-b347-4c96-b476-f3341089ff50\") " pod="openshift-ingress/e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4" Jul 28 22:49:40.660610 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:40.660583 2553 scope.go:117] "RemoveContainer" containerID="d1dc36016beec6c171b5c47e3d7416102f7c365fd49cf6c14cb96fcb3fa2171c" Jul 28 22:49:40.767181 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:40.767155 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4" Jul 28 22:49:40.908596 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:40.908567 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4"] Jul 28 22:49:40.911184 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:49:40.911158 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdd68e6b_b347_4c96_b476_f3341089ff50.slice/crio-75b271c2e0e6f1c4b97a761a4ab134ede2a8c3976bd6da9d61451fa964da4405 WatchSource:0}: Error finding container 75b271c2e0e6f1c4b97a761a4ab134ede2a8c3976bd6da9d61451fa964da4405: Status 404 returned error can't find the container with id 75b271c2e0e6f1c4b97a761a4ab134ede2a8c3976bd6da9d61451fa964da4405 Jul 28 22:49:40.913484 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:40.913454 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:49:40.913562 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:40.913523 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:49:40.913562 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:40.913551 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:49:41.086387 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:41.086287 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4" event={"ID":"cdd68e6b-b347-4c96-b476-f3341089ff50","Type":"ContainerStarted","Data":"72b9a1d2a5f85a23efa8de64794dd6071ebb0f4858c755991a6096f945b97850"} Jul 28 22:49:41.086387 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:41.086349 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4" event={"ID":"cdd68e6b-b347-4c96-b476-f3341089ff50","Type":"ContainerStarted","Data":"75b271c2e0e6f1c4b97a761a4ab134ede2a8c3976bd6da9d61451fa964da4405"} Jul 28 22:49:41.104302 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:41.104250 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4" podStartSLOduration=1.10423521 podStartE2EDuration="1.10423521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:49:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:49:41.10227922 +0000 UTC m=+2401.189537301" watchObservedRunningTime="2026-07-28 22:49:41.10423521 +0000 UTC m=+2401.191493278" Jul 28 22:49:41.767333 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:41.767302 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4" Jul 28 22:49:41.772671 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:41.772646 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4" Jul 28 22:49:42.091799 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:42.091717 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4" Jul 28 22:49:42.092612 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:42.092593 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4" Jul 28 22:49:58.132864 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:58.132820 2553 patch_prober.go:28] interesting pod/e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 28 22:49:58.133294 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:58.132921 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4" podUID="cdd68e6b-b347-4c96-b476-f3341089ff50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 28 22:49:59.185539 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:59.185506 2553 generic.go:356] "Generic (PLEG): container finished" podID="cdd68e6b-b347-4c96-b476-f3341089ff50" containerID="72b9a1d2a5f85a23efa8de64794dd6071ebb0f4858c755991a6096f945b97850" exitCode=0 Jul 28 22:49:59.185974 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:59.185585 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4" event={"ID":"cdd68e6b-b347-4c96-b476-f3341089ff50","Type":"ContainerDied","Data":"72b9a1d2a5f85a23efa8de64794dd6071ebb0f4858c755991a6096f945b97850"} Jul 28 22:49:59.186074 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:59.186055 2553 scope.go:117] "RemoveContainer" containerID="72b9a1d2a5f85a23efa8de64794dd6071ebb0f4858c755991a6096f945b97850" Jul 28 22:49:59.187264 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:59.187238 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:49:59.187335 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:59.187315 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:49:59.187376 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:49:59.187347 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:50:00.195770 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:50:00.195735 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4" event={"ID":"cdd68e6b-b347-4c96-b476-f3341089ff50","Type":"ContainerStarted","Data":"80d4820ac8d4583e45bd69cba29aa2b9495f142ca7e30bcdc190f2fc9f679f30"} Jul 28 22:50:00.768033 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:50:00.767947 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4" Jul 28 22:50:00.768198 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:50:00.768141 2553 patch_prober.go:28] interesting pod/e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Jul 28 22:50:00.768198 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:50:00.768189 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4" podUID="cdd68e6b-b347-4c96-b476-f3341089ff50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Jul 28 22:50:01.768051 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:50:01.768015 2553 patch_prober.go:28] interesting pod/e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Jul 28 22:50:01.768524 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:50:01.768070 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4" podUID="cdd68e6b-b347-4c96-b476-f3341089ff50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Jul 28 22:50:02.768238 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:50:02.768203 2553 patch_prober.go:28] interesting pod/e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Jul 28 22:50:02.768633 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:50:02.768276 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4" podUID="cdd68e6b-b347-4c96-b476-f3341089ff50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Jul 28 22:50:03.768576 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:50:03.768533 2553 patch_prober.go:28] interesting pod/e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Jul 28 22:50:03.769005 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:50:03.768595 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4" podUID="cdd68e6b-b347-4c96-b476-f3341089ff50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Jul 28 22:50:04.771633 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:50:04.771597 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4" Jul 28 22:50:04.772104 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:50:04.772052 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4" Jul 28 22:50:04.772836 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:50:04.772818 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4" Jul 28 22:53:40.717214 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:53:40.717180 2553 scope.go:117] "RemoveContainer" containerID="01fe2f4e036daf2b7fc7b20572c6f7b6d179b2f3717e3626bfe34ac2895f7a73" Jul 28 22:54:40.562120 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:54:40.562028 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mldhp_e1ced996-f136-4d3e-9c85-a5b7809b4517/console-operator/1.log" Jul 28 22:54:40.575666 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:54:40.575642 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mldhp_e1ced996-f136-4d3e-9c85-a5b7809b4517/console-operator/1.log" Jul 28 22:59:05.617490 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:05.617453 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn"] Jul 28 22:59:05.621006 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:05.620990 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn" Jul 28 22:59:05.624588 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:05.624566 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-c6c731-openshift-default-dockercfg-xtjpm\"" Jul 28 22:59:05.639504 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:05.639458 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn"] Jul 28 22:59:05.719822 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:05.719781 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17-istio-data\") pod \"e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn\" (UID: \"848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17\") " pod="openshift-ingress/e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn" Jul 28 22:59:05.720013 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:05.719833 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17-istio-podinfo\") pod \"e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn\" (UID: \"848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17\") " pod="openshift-ingress/e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn" Jul 28 22:59:05.720013 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:05.719906 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17-workload-certs\") pod \"e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn\" (UID: \"848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17\") " pod="openshift-ingress/e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn" Jul 28 22:59:05.720013 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:05.719930 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17-istio-envoy\") pod \"e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn\" (UID: \"848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17\") " pod="openshift-ingress/e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn" Jul 28 22:59:05.720013 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:05.719991 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17-credential-socket\") pod \"e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn\" (UID: \"848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17\") " pod="openshift-ingress/e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn" Jul 28 22:59:05.720227 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:05.720066 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btnjt\" (UniqueName: \"kubernetes.io/projected/848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17-kube-api-access-btnjt\") pod \"e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn\" (UID: \"848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17\") " pod="openshift-ingress/e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn" Jul 28 22:59:05.720227 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:05.720135 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17-workload-socket\") pod \"e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn\" (UID: \"848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17\") " pod="openshift-ingress/e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn" Jul 28 22:59:05.720227 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:05.720184 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17-istiod-ca-cert\") pod \"e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn\" (UID: \"848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17\") " pod="openshift-ingress/e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn" Jul 28 22:59:05.720227 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:05.720217 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17-istio-token\") pod \"e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn\" (UID: \"848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17\") " pod="openshift-ingress/e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn" Jul 28 22:59:05.821294 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:05.821244 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17-credential-socket\") pod \"e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn\" (UID: \"848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17\") " pod="openshift-ingress/e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn" Jul 28 22:59:05.821294 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:05.821301 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btnjt\" (UniqueName: \"kubernetes.io/projected/848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17-kube-api-access-btnjt\") pod \"e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn\" (UID: \"848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17\") " pod="openshift-ingress/e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn" Jul 28 22:59:05.821566 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:05.821334 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17-workload-socket\") pod \"e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn\" (UID: \"848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17\") " pod="openshift-ingress/e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn" Jul 28 22:59:05.821566 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:05.821355 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17-istiod-ca-cert\") pod \"e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn\" (UID: \"848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17\") " pod="openshift-ingress/e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn" Jul 28 22:59:05.821566 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:05.821378 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17-istio-token\") pod \"e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn\" (UID: \"848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17\") " pod="openshift-ingress/e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn" Jul 28 22:59:05.821566 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:05.821412 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17-istio-data\") pod \"e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn\" (UID: \"848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17\") " pod="openshift-ingress/e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn" Jul 28 22:59:05.821566 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:05.821429 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17-istio-podinfo\") pod \"e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn\" (UID: \"848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17\") " pod="openshift-ingress/e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn" Jul 28 22:59:05.821566 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:05.821515 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17-workload-certs\") pod \"e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn\" (UID: \"848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17\") " pod="openshift-ingress/e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn" Jul 28 22:59:05.821566 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:05.821548 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17-istio-envoy\") pod \"e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn\" (UID: \"848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17\") " pod="openshift-ingress/e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn" Jul 28 22:59:05.821967 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:05.821866 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17-workload-socket\") pod \"e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn\" (UID: \"848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17\") " pod="openshift-ingress/e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn" Jul 28 22:59:05.821967 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:05.821907 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17-workload-certs\") pod \"e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn\" (UID: \"848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17\") " pod="openshift-ingress/e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn" Jul 28 22:59:05.821967 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:05.821963 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17-credential-socket\") pod \"e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn\" (UID: \"848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17\") " pod="openshift-ingress/e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn" Jul 28 22:59:05.822140 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:05.822012 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17-istio-data\") pod \"e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn\" (UID: \"848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17\") " pod="openshift-ingress/e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn" Jul 28 22:59:05.822272 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:05.822250 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17-istiod-ca-cert\") pod \"e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn\" (UID: \"848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17\") " pod="openshift-ingress/e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn" Jul 28 22:59:05.823929 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:05.823910 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17-istio-envoy\") pod \"e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn\" (UID: \"848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17\") " pod="openshift-ingress/e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn" Jul 28 22:59:05.824159 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:05.824142 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17-istio-podinfo\") pod \"e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn\" (UID: \"848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17\") " pod="openshift-ingress/e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn" Jul 28 22:59:05.828639 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:05.828611 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17-istio-token\") pod \"e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn\" (UID: \"848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17\") " pod="openshift-ingress/e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn" Jul 28 22:59:05.828734 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:05.828698 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btnjt\" (UniqueName: \"kubernetes.io/projected/848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17-kube-api-access-btnjt\") pod \"e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn\" (UID: \"848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17\") " pod="openshift-ingress/e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn" Jul 28 22:59:05.931995 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:05.931912 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn" Jul 28 22:59:06.076901 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:06.076874 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn"] Jul 28 22:59:06.079620 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:59:06.079594 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod848b7fe8_0ad6_49ea_9fb8_1ce2d2e46f17.slice/crio-c3182e0a83d85d43846ef476c2cac8e62c4f037099894c2a28fffa831fb31909 WatchSource:0}: Error finding container c3182e0a83d85d43846ef476c2cac8e62c4f037099894c2a28fffa831fb31909: Status 404 returned error can't find the container with id c3182e0a83d85d43846ef476c2cac8e62c4f037099894c2a28fffa831fb31909 Jul 28 22:59:06.082159 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:06.082128 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:59:06.082272 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:06.082205 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:59:06.082337 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:06.082272 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:59:06.093157 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:06.093135 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn" event={"ID":"848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17","Type":"ContainerStarted","Data":"c3182e0a83d85d43846ef476c2cac8e62c4f037099894c2a28fffa831fb31909"} Jul 28 22:59:07.099779 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:07.099738 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn" event={"ID":"848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17","Type":"ContainerStarted","Data":"0be8f0e240297bba8fdaad38e2de09fbb8797617cbc31c76537d1cbfc65ff30e"} Jul 28 22:59:07.120003 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:07.119958 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn" podStartSLOduration=2.119938554 podStartE2EDuration="2.119938554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:59:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:59:07.118788415 +0000 UTC m=+2967.206046483" watchObservedRunningTime="2026-07-28 22:59:07.119938554 +0000 UTC m=+2967.207196624" Jul 28 22:59:07.932539 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:07.932499 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn" Jul 28 22:59:07.937581 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:07.937555 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn" Jul 28 22:59:08.105300 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:08.105263 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn" Jul 28 22:59:08.106162 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:08.106145 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn" Jul 28 22:59:09.215857 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:09.215819 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6c731-559f5fd57b-k6xxd"] Jul 28 22:59:09.219751 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:09.219726 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6c731-559f5fd57b-k6xxd" Jul 28 22:59:09.222151 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:09.222127 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-c6c731-serving-cert\"" Jul 28 22:59:09.229794 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:09.229558 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6c731-559f5fd57b-k6xxd"] Jul 28 22:59:09.255459 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:09.255427 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4896\" (UniqueName: \"kubernetes.io/projected/72093d06-b1c0-4bb2-be08-dbf79e6d8d2f-kube-api-access-f4896\") pod \"maas-api-e2e-ipp-a-c6c731-559f5fd57b-k6xxd\" (UID: \"72093d06-b1c0-4bb2-be08-dbf79e6d8d2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6c731-559f5fd57b-k6xxd" Jul 28 22:59:09.255629 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:09.255508 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72093d06-b1c0-4bb2-be08-dbf79e6d8d2f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c6c731-559f5fd57b-k6xxd\" (UID: \"72093d06-b1c0-4bb2-be08-dbf79e6d8d2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6c731-559f5fd57b-k6xxd" Jul 28 22:59:09.255629 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:09.255570 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72093d06-b1c0-4bb2-be08-dbf79e6d8d2f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c6c731-559f5fd57b-k6xxd\" (UID: \"72093d06-b1c0-4bb2-be08-dbf79e6d8d2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6c731-559f5fd57b-k6xxd" Jul 28 22:59:09.357090 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:09.357056 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4896\" (UniqueName: \"kubernetes.io/projected/72093d06-b1c0-4bb2-be08-dbf79e6d8d2f-kube-api-access-f4896\") pod \"maas-api-e2e-ipp-a-c6c731-559f5fd57b-k6xxd\" (UID: \"72093d06-b1c0-4bb2-be08-dbf79e6d8d2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6c731-559f5fd57b-k6xxd" Jul 28 22:59:09.357268 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:09.357135 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72093d06-b1c0-4bb2-be08-dbf79e6d8d2f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c6c731-559f5fd57b-k6xxd\" (UID: \"72093d06-b1c0-4bb2-be08-dbf79e6d8d2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6c731-559f5fd57b-k6xxd" Jul 28 22:59:09.357268 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:09.357159 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72093d06-b1c0-4bb2-be08-dbf79e6d8d2f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c6c731-559f5fd57b-k6xxd\" (UID: \"72093d06-b1c0-4bb2-be08-dbf79e6d8d2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6c731-559f5fd57b-k6xxd" Jul 28 22:59:09.357352 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:59:09.357276 2553 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6c731-serving-cert: secret "maas-api-e2e-ipp-a-c6c731-serving-cert" not found Jul 28 22:59:09.357352 ip-10-0-142-132 kubenswrapper[2553]: E0728 22:59:09.357340 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72093d06-b1c0-4bb2-be08-dbf79e6d8d2f-maas-api-tls podName:72093d06-b1c0-4bb2-be08-dbf79e6d8d2f nodeName:}" failed. No retries permitted until 2026-07-28 22:59:09.857324528 +0000 UTC m=+2969.944582575 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/72093d06-b1c0-4bb2-be08-dbf79e6d8d2f-maas-api-tls") pod "maas-api-e2e-ipp-a-c6c731-559f5fd57b-k6xxd" (UID: "72093d06-b1c0-4bb2-be08-dbf79e6d8d2f") : secret "maas-api-e2e-ipp-a-c6c731-serving-cert" not found Jul 28 22:59:09.357804 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:09.357789 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72093d06-b1c0-4bb2-be08-dbf79e6d8d2f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c6c731-559f5fd57b-k6xxd\" (UID: \"72093d06-b1c0-4bb2-be08-dbf79e6d8d2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6c731-559f5fd57b-k6xxd" Jul 28 22:59:09.366794 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:09.366770 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4896\" (UniqueName: \"kubernetes.io/projected/72093d06-b1c0-4bb2-be08-dbf79e6d8d2f-kube-api-access-f4896\") pod \"maas-api-e2e-ipp-a-c6c731-559f5fd57b-k6xxd\" (UID: \"72093d06-b1c0-4bb2-be08-dbf79e6d8d2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6c731-559f5fd57b-k6xxd" Jul 28 22:59:09.863023 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:09.862988 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72093d06-b1c0-4bb2-be08-dbf79e6d8d2f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c6c731-559f5fd57b-k6xxd\" (UID: \"72093d06-b1c0-4bb2-be08-dbf79e6d8d2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6c731-559f5fd57b-k6xxd" Jul 28 22:59:09.865756 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:09.865733 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72093d06-b1c0-4bb2-be08-dbf79e6d8d2f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c6c731-559f5fd57b-k6xxd\" (UID: \"72093d06-b1c0-4bb2-be08-dbf79e6d8d2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6c731-559f5fd57b-k6xxd" Jul 28 22:59:10.132736 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:10.132634 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6c731-559f5fd57b-k6xxd" Jul 28 22:59:10.274041 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:10.274013 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6c731-559f5fd57b-k6xxd"] Jul 28 22:59:10.275628 ip-10-0-142-132 kubenswrapper[2553]: W0728 22:59:10.275599 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72093d06_b1c0_4bb2_be08_dbf79e6d8d2f.slice/crio-5f8e195291aca6cb78c5a51aeb410f9b5b4747fdcd32f530f70431e740cc4f50 WatchSource:0}: Error finding container 5f8e195291aca6cb78c5a51aeb410f9b5b4747fdcd32f530f70431e740cc4f50: Status 404 returned error can't find the container with id 5f8e195291aca6cb78c5a51aeb410f9b5b4747fdcd32f530f70431e740cc4f50 Jul 28 22:59:10.277642 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:10.277619 2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 22:59:11.125935 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:11.125849 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6c731-559f5fd57b-k6xxd" event={"ID":"72093d06-b1c0-4bb2-be08-dbf79e6d8d2f","Type":"ContainerStarted","Data":"31337fcd4b61db326099cef3254156728440f87d92a699354242953547c06acb"} Jul 28 22:59:11.125935 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:11.125905 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6c731-559f5fd57b-k6xxd" event={"ID":"72093d06-b1c0-4bb2-be08-dbf79e6d8d2f","Type":"ContainerStarted","Data":"5f8e195291aca6cb78c5a51aeb410f9b5b4747fdcd32f530f70431e740cc4f50"} Jul 28 22:59:11.125935 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:11.125925 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6c731-559f5fd57b-k6xxd" Jul 28 22:59:11.143528 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:11.143484 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6c731-559f5fd57b-k6xxd" podStartSLOduration=1.631853005 podStartE2EDuration="2.143469955s" totalImagesPullingTime="511.61695ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 22:59:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 22:59:11.143094202 +0000 UTC m=+2971.230352272" watchObservedRunningTime="2026-07-28 22:59:11.143469955 +0000 UTC m=+2971.230728052" Jul 28 22:59:17.137033 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:17.137005 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6c731-559f5fd57b-k6xxd" Jul 28 22:59:26.207963 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:26.207874 2553 generic.go:356] "Generic (PLEG): container finished" podID="848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17" containerID="0be8f0e240297bba8fdaad38e2de09fbb8797617cbc31c76537d1cbfc65ff30e" exitCode=0 Jul 28 22:59:26.208367 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:26.207961 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn" event={"ID":"848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17","Type":"ContainerDied","Data":"0be8f0e240297bba8fdaad38e2de09fbb8797617cbc31c76537d1cbfc65ff30e"} Jul 28 22:59:26.208462 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:26.208444 2553 scope.go:117] "RemoveContainer" containerID="0be8f0e240297bba8fdaad38e2de09fbb8797617cbc31c76537d1cbfc65ff30e" Jul 28 22:59:26.209514 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:26.209489 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:59:26.209566 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:26.209556 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:59:26.209607 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:26.209587 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 22:59:27.219131 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:27.219086 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn" event={"ID":"848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17","Type":"ContainerStarted","Data":"3ebbb3e99a6fbc340242bae11a276d64e9ac56492e5a6258db1789c7b0aa4929"} Jul 28 22:59:27.932648 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:27.932600 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn" Jul 28 22:59:27.932895 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:27.932765 2553 patch_prober.go:28] interesting pod/e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Jul 28 22:59:27.932895 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:27.932808 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn" podUID="848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Jul 28 22:59:28.933424 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:28.933374 2553 patch_prober.go:28] interesting pod/e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Jul 28 22:59:28.933995 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:28.933448 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn" podUID="848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Jul 28 22:59:29.933271 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:29.933231 2553 patch_prober.go:28] interesting pod/e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Jul 28 22:59:29.933594 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:29.933552 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn" podUID="848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Jul 28 22:59:30.932971 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:30.932927 2553 patch_prober.go:28] interesting pod/e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Jul 28 22:59:30.933301 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:30.933270 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn" podUID="848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Jul 28 22:59:31.936498 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:31.936460 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn" Jul 28 22:59:31.936910 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:31.936731 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn" Jul 28 22:59:31.937422 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:31.937406 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn" Jul 28 22:59:40.598039 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:40.598004 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mldhp_e1ced996-f136-4d3e-9c85-a5b7809b4517/console-operator/1.log" Jul 28 22:59:40.612459 ip-10-0-142-132 kubenswrapper[2553]: I0728 22:59:40.612433 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mldhp_e1ced996-f136-4d3e-9c85-a5b7809b4517/console-operator/1.log" Jul 28 23:01:03.060504 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:03.060437 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk"] Jul 28 23:01:03.064210 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:03.064190 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk" Jul 28 23:01:03.066268 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:03.066246 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-030bcd-openshift-default-dockercfg-jx2jl\"" Jul 28 23:01:03.074509 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:03.074491 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk"] Jul 28 23:01:03.152627 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:03.152564 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bb95084-bc1c-4a63-87c2-8f9594d8fb4a-istio-podinfo\") pod \"e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk\" (UID: \"0bb95084-bc1c-4a63-87c2-8f9594d8fb4a\") " pod="openshift-ingress/e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk" Jul 28 23:01:03.152852 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:03.152646 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb95084-bc1c-4a63-87c2-8f9594d8fb4a-credential-socket\") pod \"e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk\" (UID: \"0bb95084-bc1c-4a63-87c2-8f9594d8fb4a\") " pod="openshift-ingress/e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk" Jul 28 23:01:03.152852 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:03.152761 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bb95084-bc1c-4a63-87c2-8f9594d8fb4a-istio-envoy\") pod \"e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk\" (UID: \"0bb95084-bc1c-4a63-87c2-8f9594d8fb4a\") " pod="openshift-ingress/e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk" Jul 28 23:01:03.152852 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:03.152807 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bb95084-bc1c-4a63-87c2-8f9594d8fb4a-istiod-ca-cert\") pod \"e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk\" (UID: \"0bb95084-bc1c-4a63-87c2-8f9594d8fb4a\") " pod="openshift-ingress/e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk" Jul 28 23:01:03.153050 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:03.152852 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bb95084-bc1c-4a63-87c2-8f9594d8fb4a-workload-certs\") pod \"e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk\" (UID: \"0bb95084-bc1c-4a63-87c2-8f9594d8fb4a\") " pod="openshift-ingress/e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk" Jul 28 23:01:03.153050 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:03.152898 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb95084-bc1c-4a63-87c2-8f9594d8fb4a-workload-socket\") pod \"e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk\" (UID: \"0bb95084-bc1c-4a63-87c2-8f9594d8fb4a\") " pod="openshift-ingress/e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk" Jul 28 23:01:03.153050 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:03.152929 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpzmz\" (UniqueName: \"kubernetes.io/projected/0bb95084-bc1c-4a63-87c2-8f9594d8fb4a-kube-api-access-bpzmz\") pod \"e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk\" (UID: \"0bb95084-bc1c-4a63-87c2-8f9594d8fb4a\") " pod="openshift-ingress/e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk" Jul 28 23:01:03.153050 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:03.152955 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bb95084-bc1c-4a63-87c2-8f9594d8fb4a-istio-data\") pod \"e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk\" (UID: \"0bb95084-bc1c-4a63-87c2-8f9594d8fb4a\") " pod="openshift-ingress/e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk" Jul 28 23:01:03.153050 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:03.152982 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bb95084-bc1c-4a63-87c2-8f9594d8fb4a-istio-token\") pod \"e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk\" (UID: \"0bb95084-bc1c-4a63-87c2-8f9594d8fb4a\") " pod="openshift-ingress/e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk" Jul 28 23:01:03.254167 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:03.254129 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bb95084-bc1c-4a63-87c2-8f9594d8fb4a-istio-envoy\") pod \"e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk\" (UID: \"0bb95084-bc1c-4a63-87c2-8f9594d8fb4a\") " pod="openshift-ingress/e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk" Jul 28 23:01:03.254359 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:03.254178 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bb95084-bc1c-4a63-87c2-8f9594d8fb4a-istiod-ca-cert\") pod \"e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk\" (UID: \"0bb95084-bc1c-4a63-87c2-8f9594d8fb4a\") " pod="openshift-ingress/e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk" Jul 28 23:01:03.254359 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:03.254206 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bb95084-bc1c-4a63-87c2-8f9594d8fb4a-workload-certs\") pod \"e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk\" (UID: \"0bb95084-bc1c-4a63-87c2-8f9594d8fb4a\") " pod="openshift-ingress/e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk" Jul 28 23:01:03.254359 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:03.254232 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb95084-bc1c-4a63-87c2-8f9594d8fb4a-workload-socket\") pod \"e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk\" (UID: \"0bb95084-bc1c-4a63-87c2-8f9594d8fb4a\") " pod="openshift-ingress/e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk" Jul 28 23:01:03.254359 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:03.254253 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpzmz\" (UniqueName: \"kubernetes.io/projected/0bb95084-bc1c-4a63-87c2-8f9594d8fb4a-kube-api-access-bpzmz\") pod \"e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk\" (UID: \"0bb95084-bc1c-4a63-87c2-8f9594d8fb4a\") " pod="openshift-ingress/e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk" Jul 28 23:01:03.254359 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:03.254276 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bb95084-bc1c-4a63-87c2-8f9594d8fb4a-istio-data\") pod \"e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk\" (UID: \"0bb95084-bc1c-4a63-87c2-8f9594d8fb4a\") " pod="openshift-ingress/e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk" Jul 28 23:01:03.254359 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:03.254300 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bb95084-bc1c-4a63-87c2-8f9594d8fb4a-istio-token\") pod \"e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk\" (UID: \"0bb95084-bc1c-4a63-87c2-8f9594d8fb4a\") " pod="openshift-ingress/e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk" Jul 28 23:01:03.254359 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:03.254349 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bb95084-bc1c-4a63-87c2-8f9594d8fb4a-istio-podinfo\") pod \"e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk\" (UID: \"0bb95084-bc1c-4a63-87c2-8f9594d8fb4a\") " pod="openshift-ingress/e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk" Jul 28 23:01:03.254752 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:03.254377 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb95084-bc1c-4a63-87c2-8f9594d8fb4a-credential-socket\") pod \"e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk\" (UID: \"0bb95084-bc1c-4a63-87c2-8f9594d8fb4a\") " pod="openshift-ingress/e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk" Jul 28 23:01:03.254752 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:03.254661 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bb95084-bc1c-4a63-87c2-8f9594d8fb4a-workload-certs\") pod \"e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk\" (UID: \"0bb95084-bc1c-4a63-87c2-8f9594d8fb4a\") " pod="openshift-ingress/e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk" Jul 28 23:01:03.254752 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:03.254717 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb95084-bc1c-4a63-87c2-8f9594d8fb4a-workload-socket\") pod \"e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk\" (UID: \"0bb95084-bc1c-4a63-87c2-8f9594d8fb4a\") " pod="openshift-ingress/e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk" Jul 28 23:01:03.254926 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:03.254900 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bb95084-bc1c-4a63-87c2-8f9594d8fb4a-istio-data\") pod \"e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk\" (UID: \"0bb95084-bc1c-4a63-87c2-8f9594d8fb4a\") " pod="openshift-ingress/e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk" Jul 28 23:01:03.254992 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:03.254948 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb95084-bc1c-4a63-87c2-8f9594d8fb4a-credential-socket\") pod \"e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk\" (UID: \"0bb95084-bc1c-4a63-87c2-8f9594d8fb4a\") " pod="openshift-ingress/e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk" Jul 28 23:01:03.255114 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:03.255095 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bb95084-bc1c-4a63-87c2-8f9594d8fb4a-istiod-ca-cert\") pod \"e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk\" (UID: \"0bb95084-bc1c-4a63-87c2-8f9594d8fb4a\") " pod="openshift-ingress/e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk" Jul 28 23:01:03.256550 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:03.256533 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bb95084-bc1c-4a63-87c2-8f9594d8fb4a-istio-envoy\") pod \"e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk\" (UID: \"0bb95084-bc1c-4a63-87c2-8f9594d8fb4a\") " pod="openshift-ingress/e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk" Jul 28 23:01:03.257136 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:03.257112 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bb95084-bc1c-4a63-87c2-8f9594d8fb4a-istio-podinfo\") pod \"e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk\" (UID: \"0bb95084-bc1c-4a63-87c2-8f9594d8fb4a\") " pod="openshift-ingress/e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk" Jul 28 23:01:03.261497 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:03.261476 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bb95084-bc1c-4a63-87c2-8f9594d8fb4a-istio-token\") pod \"e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk\" (UID: \"0bb95084-bc1c-4a63-87c2-8f9594d8fb4a\") " pod="openshift-ingress/e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk" Jul 28 23:01:03.261630 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:03.261613 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpzmz\" (UniqueName: \"kubernetes.io/projected/0bb95084-bc1c-4a63-87c2-8f9594d8fb4a-kube-api-access-bpzmz\") pod \"e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk\" (UID: \"0bb95084-bc1c-4a63-87c2-8f9594d8fb4a\") " pod="openshift-ingress/e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk" Jul 28 23:01:03.375536 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:03.375440 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk" Jul 28 23:01:03.539587 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:03.536966 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 23:01:03.539587 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:03.537143 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 23:01:03.539587 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:03.537228 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 23:01:03.543507 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:03.543477 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk"] Jul 28 23:01:03.776762 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:03.776696 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk" event={"ID":"0bb95084-bc1c-4a63-87c2-8f9594d8fb4a","Type":"ContainerStarted","Data":"32f0fa6b8e4a73f3e96ee4e653bd71882e211b0bf1b82773024a6c2ca6c2ede4"} Jul 28 23:01:03.776974 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:03.776799 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk" event={"ID":"0bb95084-bc1c-4a63-87c2-8f9594d8fb4a","Type":"ContainerStarted","Data":"094c1c90b5a3197815397fb30180e69df6c84a0a10b34c3fd242f6d46fa3e05a"} Jul 28 23:01:03.794341 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:03.794283 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk" podStartSLOduration=0.794229627 podStartE2EDuration="794.229627ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 23:01:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:01:03.793564186 +0000 UTC m=+3083.880822259" watchObservedRunningTime="2026-07-28 23:01:03.794229627 +0000 UTC m=+3083.881487699" Jul 28 23:01:04.376194 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:04.376159 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk" Jul 28 23:01:04.382819 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:04.382786 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk" Jul 28 23:01:04.782773 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:04.782741 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk" Jul 28 23:01:04.783801 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:04.783680 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk" Jul 28 23:01:06.695145 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:06.695116 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-030bcd-5fb5484bcd-cp7st"] Jul 28 23:01:06.698947 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:06.698927 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-030bcd-5fb5484bcd-cp7st" Jul 28 23:01:06.700953 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:06.700932 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-030bcd-serving-cert\"" Jul 28 23:01:06.707510 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:06.707486 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-030bcd-5fb5484bcd-cp7st"] Jul 28 23:01:06.791806 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:06.791757 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp84g\" (UniqueName: \"kubernetes.io/projected/8e5ecb88-9e83-4f8f-be96-bc5570d3cdcd-kube-api-access-kp84g\") pod \"maas-api-e2e-ipp-cleanup-030bcd-5fb5484bcd-cp7st\" (UID: \"8e5ecb88-9e83-4f8f-be96-bc5570d3cdcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-030bcd-5fb5484bcd-cp7st" Jul 28 23:01:06.792250 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:06.792223 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e5ecb88-9e83-4f8f-be96-bc5570d3cdcd-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-030bcd-5fb5484bcd-cp7st\" (UID: \"8e5ecb88-9e83-4f8f-be96-bc5570d3cdcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-030bcd-5fb5484bcd-cp7st" Jul 28 23:01:06.792499 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:06.792480 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e5ecb88-9e83-4f8f-be96-bc5570d3cdcd-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-030bcd-5fb5484bcd-cp7st\" (UID: \"8e5ecb88-9e83-4f8f-be96-bc5570d3cdcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-030bcd-5fb5484bcd-cp7st" Jul 28 23:01:06.894181 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:06.894083 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e5ecb88-9e83-4f8f-be96-bc5570d3cdcd-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-030bcd-5fb5484bcd-cp7st\" (UID: \"8e5ecb88-9e83-4f8f-be96-bc5570d3cdcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-030bcd-5fb5484bcd-cp7st" Jul 28 23:01:06.894347 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:06.894257 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp84g\" (UniqueName: \"kubernetes.io/projected/8e5ecb88-9e83-4f8f-be96-bc5570d3cdcd-kube-api-access-kp84g\") pod \"maas-api-e2e-ipp-cleanup-030bcd-5fb5484bcd-cp7st\" (UID: \"8e5ecb88-9e83-4f8f-be96-bc5570d3cdcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-030bcd-5fb5484bcd-cp7st" Jul 28 23:01:06.894439 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:06.894420 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e5ecb88-9e83-4f8f-be96-bc5570d3cdcd-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-030bcd-5fb5484bcd-cp7st\" (UID: \"8e5ecb88-9e83-4f8f-be96-bc5570d3cdcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-030bcd-5fb5484bcd-cp7st" Jul 28 23:01:06.894572 ip-10-0-142-132 kubenswrapper[2553]: E0728 23:01:06.894556 2553 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-030bcd-serving-cert: secret "maas-api-e2e-ipp-cleanup-030bcd-serving-cert" not found Jul 28 23:01:06.894637 ip-10-0-142-132 kubenswrapper[2553]: E0728 23:01:06.894625 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e5ecb88-9e83-4f8f-be96-bc5570d3cdcd-maas-api-tls podName:8e5ecb88-9e83-4f8f-be96-bc5570d3cdcd nodeName:}" failed. No retries permitted until 2026-07-28 23:01:07.394606385 +0000 UTC m=+3087.481864435 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8e5ecb88-9e83-4f8f-be96-bc5570d3cdcd-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-030bcd-5fb5484bcd-cp7st" (UID: "8e5ecb88-9e83-4f8f-be96-bc5570d3cdcd") : secret "maas-api-e2e-ipp-cleanup-030bcd-serving-cert" not found Jul 28 23:01:06.894905 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:06.894889 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e5ecb88-9e83-4f8f-be96-bc5570d3cdcd-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-030bcd-5fb5484bcd-cp7st\" (UID: \"8e5ecb88-9e83-4f8f-be96-bc5570d3cdcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-030bcd-5fb5484bcd-cp7st" Jul 28 23:01:06.904134 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:06.904105 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp84g\" (UniqueName: \"kubernetes.io/projected/8e5ecb88-9e83-4f8f-be96-bc5570d3cdcd-kube-api-access-kp84g\") pod \"maas-api-e2e-ipp-cleanup-030bcd-5fb5484bcd-cp7st\" (UID: \"8e5ecb88-9e83-4f8f-be96-bc5570d3cdcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-030bcd-5fb5484bcd-cp7st" Jul 28 23:01:07.399612 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:07.399578 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e5ecb88-9e83-4f8f-be96-bc5570d3cdcd-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-030bcd-5fb5484bcd-cp7st\" (UID: \"8e5ecb88-9e83-4f8f-be96-bc5570d3cdcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-030bcd-5fb5484bcd-cp7st" Jul 28 23:01:07.399809 ip-10-0-142-132 kubenswrapper[2553]: E0728 23:01:07.399762 2553 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-030bcd-serving-cert: secret "maas-api-e2e-ipp-cleanup-030bcd-serving-cert" not found Jul 28 23:01:07.399856 ip-10-0-142-132 kubenswrapper[2553]: E0728 23:01:07.399841 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e5ecb88-9e83-4f8f-be96-bc5570d3cdcd-maas-api-tls podName:8e5ecb88-9e83-4f8f-be96-bc5570d3cdcd nodeName:}" failed. No retries permitted until 2026-07-28 23:01:08.399819664 +0000 UTC m=+3088.487077731 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8e5ecb88-9e83-4f8f-be96-bc5570d3cdcd-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-030bcd-5fb5484bcd-cp7st" (UID: "8e5ecb88-9e83-4f8f-be96-bc5570d3cdcd") : secret "maas-api-e2e-ipp-cleanup-030bcd-serving-cert" not found Jul 28 23:01:08.411584 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:08.411555 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e5ecb88-9e83-4f8f-be96-bc5570d3cdcd-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-030bcd-5fb5484bcd-cp7st\" (UID: \"8e5ecb88-9e83-4f8f-be96-bc5570d3cdcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-030bcd-5fb5484bcd-cp7st" Jul 28 23:01:08.414030 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:08.414009 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e5ecb88-9e83-4f8f-be96-bc5570d3cdcd-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-030bcd-5fb5484bcd-cp7st\" (UID: \"8e5ecb88-9e83-4f8f-be96-bc5570d3cdcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-030bcd-5fb5484bcd-cp7st" Jul 28 23:01:08.513029 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:08.512996 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-030bcd-5fb5484bcd-cp7st" Jul 28 23:01:08.665806 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:08.665777 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-030bcd-5fb5484bcd-cp7st"] Jul 28 23:01:08.667031 ip-10-0-142-132 kubenswrapper[2553]: W0728 23:01:08.667007 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e5ecb88_9e83_4f8f_be96_bc5570d3cdcd.slice/crio-54381a5df56dd95318a1ee13de14b23f00f90c8b6c32ff41cd4b6b896b6d9c2e WatchSource:0}: Error finding container 54381a5df56dd95318a1ee13de14b23f00f90c8b6c32ff41cd4b6b896b6d9c2e: Status 404 returned error can't find the container with id 54381a5df56dd95318a1ee13de14b23f00f90c8b6c32ff41cd4b6b896b6d9c2e Jul 28 23:01:08.812071 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:08.812032 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-030bcd-5fb5484bcd-cp7st" event={"ID":"8e5ecb88-9e83-4f8f-be96-bc5570d3cdcd","Type":"ContainerStarted","Data":"54381a5df56dd95318a1ee13de14b23f00f90c8b6c32ff41cd4b6b896b6d9c2e"} Jul 28 23:01:09.819060 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:09.819020 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-030bcd-5fb5484bcd-cp7st" event={"ID":"8e5ecb88-9e83-4f8f-be96-bc5570d3cdcd","Type":"ContainerStarted","Data":"23fcdd4411b2e8a42180eded8c74c4f433431b4ec5ea4e911d77afac7807cd00"} Jul 28 23:01:09.819557 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:09.819086 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-030bcd-5fb5484bcd-cp7st" Jul 28 23:01:09.834612 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:09.834566 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-030bcd-5fb5484bcd-cp7st" podStartSLOduration=3.410851487 podStartE2EDuration="3.834549993s" totalImagesPullingTime="423.698506ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 23:01:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:01:09.832901957 +0000 UTC m=+3089.920160026" watchObservedRunningTime="2026-07-28 23:01:09.834549993 +0000 UTC m=+3089.921808065" Jul 28 23:01:15.830275 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:15.830247 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-030bcd-5fb5484bcd-cp7st" Jul 28 23:01:22.894786 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:22.894751 2553 generic.go:356] "Generic (PLEG): container finished" podID="0bb95084-bc1c-4a63-87c2-8f9594d8fb4a" containerID="32f0fa6b8e4a73f3e96ee4e653bd71882e211b0bf1b82773024a6c2ca6c2ede4" exitCode=0 Jul 28 23:01:22.895204 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:22.894826 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk" event={"ID":"0bb95084-bc1c-4a63-87c2-8f9594d8fb4a","Type":"ContainerDied","Data":"32f0fa6b8e4a73f3e96ee4e653bd71882e211b0bf1b82773024a6c2ca6c2ede4"} Jul 28 23:01:22.895289 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:22.895265 2553 scope.go:117] "RemoveContainer" containerID="32f0fa6b8e4a73f3e96ee4e653bd71882e211b0bf1b82773024a6c2ca6c2ede4" Jul 28 23:01:22.896473 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:22.896434 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 23:01:22.896618 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:22.896521 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 23:01:22.896618 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:22.896588 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 28 23:01:23.902091 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:23.902056 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk" event={"ID":"0bb95084-bc1c-4a63-87c2-8f9594d8fb4a","Type":"ContainerStarted","Data":"42e02566ce3b6dde562106804a31eb0c5e9c1130bf6ee66e12fb8c1f8c68708e"} Jul 28 23:01:24.375892 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:24.375854 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk" Jul 28 23:01:24.376073 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:24.376047 2553 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Jul 28 23:01:24.376123 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:24.376097 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk" podUID="0bb95084-bc1c-4a63-87c2-8f9594d8fb4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Jul 28 23:01:25.376379 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:25.376343 2553 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Jul 28 23:01:25.376794 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:25.376410 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk" podUID="0bb95084-bc1c-4a63-87c2-8f9594d8fb4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Jul 28 23:01:26.376873 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:26.376829 2553 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Jul 28 23:01:26.377266 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:26.376891 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk" podUID="0bb95084-bc1c-4a63-87c2-8f9594d8fb4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Jul 28 23:01:27.376586 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:27.376553 2553 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Jul 28 23:01:27.376770 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:27.376609 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk" podUID="0bb95084-bc1c-4a63-87c2-8f9594d8fb4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Jul 28 23:01:28.379789 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:28.379757 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk" Jul 28 23:01:28.380213 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:28.380031 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk" Jul 28 23:01:28.380804 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:28.380787 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk" Jul 28 23:01:28.530482 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:28.530453 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-030bcd-5fb5484bcd-cp7st"] Jul 28 23:01:28.530732 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:28.530687 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-030bcd-5fb5484bcd-cp7st" podUID="8e5ecb88-9e83-4f8f-be96-bc5570d3cdcd" containerName="maas-api" containerID="cri-o://23fcdd4411b2e8a42180eded8c74c4f433431b4ec5ea4e911d77afac7807cd00" gracePeriod=30 Jul 28 23:01:28.807605 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:28.807582 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-030bcd-5fb5484bcd-cp7st" Jul 28 23:01:28.935803 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:28.935686 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kp84g\" (UniqueName: \"kubernetes.io/projected/8e5ecb88-9e83-4f8f-be96-bc5570d3cdcd-kube-api-access-kp84g\") pod \"8e5ecb88-9e83-4f8f-be96-bc5570d3cdcd\" (UID: \"8e5ecb88-9e83-4f8f-be96-bc5570d3cdcd\") " Jul 28 23:01:28.935803 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:28.935772 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e5ecb88-9e83-4f8f-be96-bc5570d3cdcd-openshift-service-ca\") pod \"8e5ecb88-9e83-4f8f-be96-bc5570d3cdcd\" (UID: \"8e5ecb88-9e83-4f8f-be96-bc5570d3cdcd\") " Jul 28 23:01:28.936044 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:28.935817 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e5ecb88-9e83-4f8f-be96-bc5570d3cdcd-maas-api-tls\") pod \"8e5ecb88-9e83-4f8f-be96-bc5570d3cdcd\" (UID: \"8e5ecb88-9e83-4f8f-be96-bc5570d3cdcd\") " Jul 28 23:01:28.936188 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:28.936165 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e5ecb88-9e83-4f8f-be96-bc5570d3cdcd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8e5ecb88-9e83-4f8f-be96-bc5570d3cdcd" (UID: "8e5ecb88-9e83-4f8f-be96-bc5570d3cdcd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 23:01:28.936262 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:28.936224 2553 generic.go:356] "Generic (PLEG): container finished" podID="8e5ecb88-9e83-4f8f-be96-bc5570d3cdcd" containerID="23fcdd4411b2e8a42180eded8c74c4f433431b4ec5ea4e911d77afac7807cd00" exitCode=0 Jul 28 23:01:28.936335 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:28.936285 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-030bcd-5fb5484bcd-cp7st" Jul 28 23:01:28.936421 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:28.936316 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-030bcd-5fb5484bcd-cp7st" event={"ID":"8e5ecb88-9e83-4f8f-be96-bc5570d3cdcd","Type":"ContainerDied","Data":"23fcdd4411b2e8a42180eded8c74c4f433431b4ec5ea4e911d77afac7807cd00"} Jul 28 23:01:28.936465 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:28.936456 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-030bcd-5fb5484bcd-cp7st" event={"ID":"8e5ecb88-9e83-4f8f-be96-bc5570d3cdcd","Type":"ContainerDied","Data":"54381a5df56dd95318a1ee13de14b23f00f90c8b6c32ff41cd4b6b896b6d9c2e"} Jul 28 23:01:28.936501 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:28.936479 2553 scope.go:117] "RemoveContainer" containerID="23fcdd4411b2e8a42180eded8c74c4f433431b4ec5ea4e911d77afac7807cd00" Jul 28 23:01:28.938310 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:28.938281 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e5ecb88-9e83-4f8f-be96-bc5570d3cdcd-kube-api-access-kp84g" (OuterVolumeSpecName: "kube-api-access-kp84g") pod "8e5ecb88-9e83-4f8f-be96-bc5570d3cdcd" (UID: "8e5ecb88-9e83-4f8f-be96-bc5570d3cdcd"). InnerVolumeSpecName "kube-api-access-kp84g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 23:01:28.938310 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:28.938295 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e5ecb88-9e83-4f8f-be96-bc5570d3cdcd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8e5ecb88-9e83-4f8f-be96-bc5570d3cdcd" (UID: "8e5ecb88-9e83-4f8f-be96-bc5570d3cdcd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 23:01:28.958384 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:28.958361 2553 scope.go:117] "RemoveContainer" containerID="23fcdd4411b2e8a42180eded8c74c4f433431b4ec5ea4e911d77afac7807cd00" Jul 28 23:01:28.958717 ip-10-0-142-132 kubenswrapper[2553]: E0728 23:01:28.958683 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23fcdd4411b2e8a42180eded8c74c4f433431b4ec5ea4e911d77afac7807cd00\": container with ID starting with 23fcdd4411b2e8a42180eded8c74c4f433431b4ec5ea4e911d77afac7807cd00 not found: ID does not exist" containerID="23fcdd4411b2e8a42180eded8c74c4f433431b4ec5ea4e911d77afac7807cd00" Jul 28 23:01:28.958780 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:28.958733 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23fcdd4411b2e8a42180eded8c74c4f433431b4ec5ea4e911d77afac7807cd00"} err="failed to get container status \"23fcdd4411b2e8a42180eded8c74c4f433431b4ec5ea4e911d77afac7807cd00\": rpc error: code = NotFound desc = could not find container \"23fcdd4411b2e8a42180eded8c74c4f433431b4ec5ea4e911d77afac7807cd00\": container with ID starting with 23fcdd4411b2e8a42180eded8c74c4f433431b4ec5ea4e911d77afac7807cd00 not found: ID does not exist" Jul 28 23:01:29.037140 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:29.037097 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kp84g\" (UniqueName: \"kubernetes.io/projected/8e5ecb88-9e83-4f8f-be96-bc5570d3cdcd-kube-api-access-kp84g\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 23:01:29.037140 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:29.037134 2553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e5ecb88-9e83-4f8f-be96-bc5570d3cdcd-openshift-service-ca\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 23:01:29.037399 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:29.037150 2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e5ecb88-9e83-4f8f-be96-bc5570d3cdcd-maas-api-tls\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 23:01:29.257455 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:29.257428 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-030bcd-5fb5484bcd-cp7st"] Jul 28 23:01:29.261190 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:29.261162 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-030bcd-5fb5484bcd-cp7st"] Jul 28 23:01:30.405001 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:30.404964 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e5ecb88-9e83-4f8f-be96-bc5570d3cdcd" path="/var/lib/kubelet/pods/8e5ecb88-9e83-4f8f-be96-bc5570d3cdcd/volumes" Jul 28 23:01:35.302466 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:35.302436 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk"] Jul 28 23:01:35.302989 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:35.302643 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk" podUID="0bb95084-bc1c-4a63-87c2-8f9594d8fb4a" containerName="istio-proxy" containerID="cri-o://42e02566ce3b6dde562106804a31eb0c5e9c1130bf6ee66e12fb8c1f8c68708e" gracePeriod=30 Jul 28 23:01:40.567469 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:40.567396 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk" Jul 28 23:01:40.655481 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:40.655443 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bb95084-bc1c-4a63-87c2-8f9594d8fb4a-istiod-ca-cert\") pod \"0bb95084-bc1c-4a63-87c2-8f9594d8fb4a\" (UID: \"0bb95084-bc1c-4a63-87c2-8f9594d8fb4a\") " Jul 28 23:01:40.655677 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:40.655502 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb95084-bc1c-4a63-87c2-8f9594d8fb4a-workload-socket\") pod \"0bb95084-bc1c-4a63-87c2-8f9594d8fb4a\" (UID: \"0bb95084-bc1c-4a63-87c2-8f9594d8fb4a\") " Jul 28 23:01:40.655677 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:40.655522 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bb95084-bc1c-4a63-87c2-8f9594d8fb4a-istio-podinfo\") pod \"0bb95084-bc1c-4a63-87c2-8f9594d8fb4a\" (UID: \"0bb95084-bc1c-4a63-87c2-8f9594d8fb4a\") " Jul 28 23:01:40.655677 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:40.655560 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb95084-bc1c-4a63-87c2-8f9594d8fb4a-credential-socket\") pod \"0bb95084-bc1c-4a63-87c2-8f9594d8fb4a\" (UID: \"0bb95084-bc1c-4a63-87c2-8f9594d8fb4a\") " Jul 28 23:01:40.655677 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:40.655607 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bb95084-bc1c-4a63-87c2-8f9594d8fb4a-istio-envoy\") pod \"0bb95084-bc1c-4a63-87c2-8f9594d8fb4a\" (UID: \"0bb95084-bc1c-4a63-87c2-8f9594d8fb4a\") " Jul 28 23:01:40.655677 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:40.655630 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bpzmz\" (UniqueName: \"kubernetes.io/projected/0bb95084-bc1c-4a63-87c2-8f9594d8fb4a-kube-api-access-bpzmz\") pod \"0bb95084-bc1c-4a63-87c2-8f9594d8fb4a\" (UID: \"0bb95084-bc1c-4a63-87c2-8f9594d8fb4a\") " Jul 28 23:01:40.655677 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:40.655655 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bb95084-bc1c-4a63-87c2-8f9594d8fb4a-workload-certs\") pod \"0bb95084-bc1c-4a63-87c2-8f9594d8fb4a\" (UID: \"0bb95084-bc1c-4a63-87c2-8f9594d8fb4a\") " Jul 28 23:01:40.655677 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:40.655680 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bb95084-bc1c-4a63-87c2-8f9594d8fb4a-istio-data\") pod \"0bb95084-bc1c-4a63-87c2-8f9594d8fb4a\" (UID: \"0bb95084-bc1c-4a63-87c2-8f9594d8fb4a\") " Jul 28 23:01:40.656060 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:40.655735 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bb95084-bc1c-4a63-87c2-8f9594d8fb4a-istio-token\") pod \"0bb95084-bc1c-4a63-87c2-8f9594d8fb4a\" (UID: \"0bb95084-bc1c-4a63-87c2-8f9594d8fb4a\") " Jul 28 23:01:40.656060 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:40.655879 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bb95084-bc1c-4a63-87c2-8f9594d8fb4a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0bb95084-bc1c-4a63-87c2-8f9594d8fb4a" (UID: "0bb95084-bc1c-4a63-87c2-8f9594d8fb4a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 23:01:40.656060 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:40.655951 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bb95084-bc1c-4a63-87c2-8f9594d8fb4a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0bb95084-bc1c-4a63-87c2-8f9594d8fb4a" (UID: "0bb95084-bc1c-4a63-87c2-8f9594d8fb4a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 23:01:40.656229 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:40.656096 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bb95084-bc1c-4a63-87c2-8f9594d8fb4a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0bb95084-bc1c-4a63-87c2-8f9594d8fb4a" (UID: "0bb95084-bc1c-4a63-87c2-8f9594d8fb4a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 23:01:40.656229 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:40.656106 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bb95084-bc1c-4a63-87c2-8f9594d8fb4a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0bb95084-bc1c-4a63-87c2-8f9594d8fb4a" (UID: "0bb95084-bc1c-4a63-87c2-8f9594d8fb4a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 23:01:40.656229 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:40.656128 2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bb95084-bc1c-4a63-87c2-8f9594d8fb4a-istiod-ca-cert\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 23:01:40.656229 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:40.656147 2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb95084-bc1c-4a63-87c2-8f9594d8fb4a-workload-socket\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 23:01:40.657025 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:40.657002 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bb95084-bc1c-4a63-87c2-8f9594d8fb4a-istio-data" (OuterVolumeSpecName: "istio-data") pod "0bb95084-bc1c-4a63-87c2-8f9594d8fb4a" (UID: "0bb95084-bc1c-4a63-87c2-8f9594d8fb4a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 23:01:40.658295 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:40.658264 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bb95084-bc1c-4a63-87c2-8f9594d8fb4a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0bb95084-bc1c-4a63-87c2-8f9594d8fb4a" (UID: "0bb95084-bc1c-4a63-87c2-8f9594d8fb4a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 23:01:40.658410 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:40.658351 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bb95084-bc1c-4a63-87c2-8f9594d8fb4a-istio-token" (OuterVolumeSpecName: "istio-token") pod "0bb95084-bc1c-4a63-87c2-8f9594d8fb4a" (UID: "0bb95084-bc1c-4a63-87c2-8f9594d8fb4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 23:01:40.658650 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:40.658624 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0bb95084-bc1c-4a63-87c2-8f9594d8fb4a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0bb95084-bc1c-4a63-87c2-8f9594d8fb4a" (UID: "0bb95084-bc1c-4a63-87c2-8f9594d8fb4a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 23:01:40.658650 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:40.658641 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bb95084-bc1c-4a63-87c2-8f9594d8fb4a-kube-api-access-bpzmz" (OuterVolumeSpecName: "kube-api-access-bpzmz") pod "0bb95084-bc1c-4a63-87c2-8f9594d8fb4a" (UID: "0bb95084-bc1c-4a63-87c2-8f9594d8fb4a"). InnerVolumeSpecName "kube-api-access-bpzmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 23:01:40.757427 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:40.757395 2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bb95084-bc1c-4a63-87c2-8f9594d8fb4a-istio-podinfo\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 23:01:40.757427 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:40.757424 2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb95084-bc1c-4a63-87c2-8f9594d8fb4a-credential-socket\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 23:01:40.757427 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:40.757434 2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bb95084-bc1c-4a63-87c2-8f9594d8fb4a-istio-envoy\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 23:01:40.757664 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:40.757444 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpzmz\" (UniqueName: \"kubernetes.io/projected/0bb95084-bc1c-4a63-87c2-8f9594d8fb4a-kube-api-access-bpzmz\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 23:01:40.757664 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:40.757453 2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bb95084-bc1c-4a63-87c2-8f9594d8fb4a-workload-certs\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 23:01:40.757664 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:40.757461 2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bb95084-bc1c-4a63-87c2-8f9594d8fb4a-istio-data\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 23:01:40.757664 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:40.757468 2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bb95084-bc1c-4a63-87c2-8f9594d8fb4a-istio-token\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 23:01:41.006420 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:41.006382 2553 generic.go:356] "Generic (PLEG): container finished" podID="0bb95084-bc1c-4a63-87c2-8f9594d8fb4a" containerID="42e02566ce3b6dde562106804a31eb0c5e9c1130bf6ee66e12fb8c1f8c68708e" exitCode=0 Jul 28 23:01:41.006586 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:41.006428 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk" Jul 28 23:01:41.006586 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:41.006467 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk" event={"ID":"0bb95084-bc1c-4a63-87c2-8f9594d8fb4a","Type":"ContainerDied","Data":"42e02566ce3b6dde562106804a31eb0c5e9c1130bf6ee66e12fb8c1f8c68708e"} Jul 28 23:01:41.006586 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:41.006524 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk" event={"ID":"0bb95084-bc1c-4a63-87c2-8f9594d8fb4a","Type":"ContainerDied","Data":"094c1c90b5a3197815397fb30180e69df6c84a0a10b34c3fd242f6d46fa3e05a"} Jul 28 23:01:41.006586 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:41.006547 2553 scope.go:117] "RemoveContainer" containerID="42e02566ce3b6dde562106804a31eb0c5e9c1130bf6ee66e12fb8c1f8c68708e" Jul 28 23:01:41.017574 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:41.017550 2553 scope.go:117] "RemoveContainer" containerID="32f0fa6b8e4a73f3e96ee4e653bd71882e211b0bf1b82773024a6c2ca6c2ede4" Jul 28 23:01:41.028476 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:41.028455 2553 scope.go:117] "RemoveContainer" containerID="42e02566ce3b6dde562106804a31eb0c5e9c1130bf6ee66e12fb8c1f8c68708e" Jul 28 23:01:41.028779 ip-10-0-142-132 kubenswrapper[2553]: E0728 23:01:41.028761 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42e02566ce3b6dde562106804a31eb0c5e9c1130bf6ee66e12fb8c1f8c68708e\": container with ID starting with 42e02566ce3b6dde562106804a31eb0c5e9c1130bf6ee66e12fb8c1f8c68708e not found: ID does not exist" containerID="42e02566ce3b6dde562106804a31eb0c5e9c1130bf6ee66e12fb8c1f8c68708e" Jul 28 23:01:41.028842 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:41.028789 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42e02566ce3b6dde562106804a31eb0c5e9c1130bf6ee66e12fb8c1f8c68708e"} err="failed to get container status \"42e02566ce3b6dde562106804a31eb0c5e9c1130bf6ee66e12fb8c1f8c68708e\": rpc error: code = NotFound desc = could not find container \"42e02566ce3b6dde562106804a31eb0c5e9c1130bf6ee66e12fb8c1f8c68708e\": container with ID starting with 42e02566ce3b6dde562106804a31eb0c5e9c1130bf6ee66e12fb8c1f8c68708e not found: ID does not exist" Jul 28 23:01:41.028842 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:41.028807 2553 scope.go:117] "RemoveContainer" containerID="32f0fa6b8e4a73f3e96ee4e653bd71882e211b0bf1b82773024a6c2ca6c2ede4" Jul 28 23:01:41.029070 ip-10-0-142-132 kubenswrapper[2553]: E0728 23:01:41.029055 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32f0fa6b8e4a73f3e96ee4e653bd71882e211b0bf1b82773024a6c2ca6c2ede4\": container with ID starting with 32f0fa6b8e4a73f3e96ee4e653bd71882e211b0bf1b82773024a6c2ca6c2ede4 not found: ID does not exist" containerID="32f0fa6b8e4a73f3e96ee4e653bd71882e211b0bf1b82773024a6c2ca6c2ede4" Jul 28 23:01:41.029119 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:41.029075 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32f0fa6b8e4a73f3e96ee4e653bd71882e211b0bf1b82773024a6c2ca6c2ede4"} err="failed to get container status \"32f0fa6b8e4a73f3e96ee4e653bd71882e211b0bf1b82773024a6c2ca6c2ede4\": rpc error: code = NotFound desc = could not find container \"32f0fa6b8e4a73f3e96ee4e653bd71882e211b0bf1b82773024a6c2ca6c2ede4\": container with ID starting with 32f0fa6b8e4a73f3e96ee4e653bd71882e211b0bf1b82773024a6c2ca6c2ede4 not found: ID does not exist" Jul 28 23:01:41.029900 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:41.029879 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk"] Jul 28 23:01:41.035404 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:41.035377 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-030bcd-openshift-default-76bc6656bd-bq4hk"] Jul 28 23:01:42.405505 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:42.405464 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bb95084-bc1c-4a63-87c2-8f9594d8fb4a" path="/var/lib/kubelet/pods/0bb95084-bc1c-4a63-87c2-8f9594d8fb4a/volumes" Jul 28 23:01:53.040301 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:53.040260 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6c731-559f5fd57b-k6xxd"] Jul 28 23:01:53.040937 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:53.040509 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6c731-559f5fd57b-k6xxd" podUID="72093d06-b1c0-4bb2-be08-dbf79e6d8d2f" containerName="maas-api" containerID="cri-o://31337fcd4b61db326099cef3254156728440f87d92a699354242953547c06acb" gracePeriod=30 Jul 28 23:01:53.310675 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:53.310646 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6c731-559f5fd57b-k6xxd" Jul 28 23:01:53.383756 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:53.383726 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72093d06-b1c0-4bb2-be08-dbf79e6d8d2f-maas-api-tls\") pod \"72093d06-b1c0-4bb2-be08-dbf79e6d8d2f\" (UID: \"72093d06-b1c0-4bb2-be08-dbf79e6d8d2f\") " Jul 28 23:01:53.383937 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:53.383770 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f4896\" (UniqueName: \"kubernetes.io/projected/72093d06-b1c0-4bb2-be08-dbf79e6d8d2f-kube-api-access-f4896\") pod \"72093d06-b1c0-4bb2-be08-dbf79e6d8d2f\" (UID: \"72093d06-b1c0-4bb2-be08-dbf79e6d8d2f\") " Jul 28 23:01:53.383937 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:53.383789 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72093d06-b1c0-4bb2-be08-dbf79e6d8d2f-openshift-service-ca\") pod \"72093d06-b1c0-4bb2-be08-dbf79e6d8d2f\" (UID: \"72093d06-b1c0-4bb2-be08-dbf79e6d8d2f\") " Jul 28 23:01:53.384166 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:53.384141 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72093d06-b1c0-4bb2-be08-dbf79e6d8d2f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "72093d06-b1c0-4bb2-be08-dbf79e6d8d2f" (UID: "72093d06-b1c0-4bb2-be08-dbf79e6d8d2f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 23:01:53.386000 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:53.385978 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72093d06-b1c0-4bb2-be08-dbf79e6d8d2f-kube-api-access-f4896" (OuterVolumeSpecName: "kube-api-access-f4896") pod "72093d06-b1c0-4bb2-be08-dbf79e6d8d2f" (UID: "72093d06-b1c0-4bb2-be08-dbf79e6d8d2f"). InnerVolumeSpecName "kube-api-access-f4896". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 23:01:53.386099 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:53.386084 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72093d06-b1c0-4bb2-be08-dbf79e6d8d2f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "72093d06-b1c0-4bb2-be08-dbf79e6d8d2f" (UID: "72093d06-b1c0-4bb2-be08-dbf79e6d8d2f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 23:01:53.485472 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:53.485442 2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72093d06-b1c0-4bb2-be08-dbf79e6d8d2f-maas-api-tls\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 23:01:53.485472 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:53.485468 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4896\" (UniqueName: \"kubernetes.io/projected/72093d06-b1c0-4bb2-be08-dbf79e6d8d2f-kube-api-access-f4896\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 23:01:53.485472 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:53.485477 2553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72093d06-b1c0-4bb2-be08-dbf79e6d8d2f-openshift-service-ca\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 23:01:54.080578 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:54.080542 2553 generic.go:356] "Generic (PLEG): container finished" podID="72093d06-b1c0-4bb2-be08-dbf79e6d8d2f" containerID="31337fcd4b61db326099cef3254156728440f87d92a699354242953547c06acb" exitCode=0 Jul 28 23:01:54.081037 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:54.080594 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6c731-559f5fd57b-k6xxd" Jul 28 23:01:54.081037 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:54.080636 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6c731-559f5fd57b-k6xxd" event={"ID":"72093d06-b1c0-4bb2-be08-dbf79e6d8d2f","Type":"ContainerDied","Data":"31337fcd4b61db326099cef3254156728440f87d92a699354242953547c06acb"} Jul 28 23:01:54.081037 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:54.080690 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6c731-559f5fd57b-k6xxd" event={"ID":"72093d06-b1c0-4bb2-be08-dbf79e6d8d2f","Type":"ContainerDied","Data":"5f8e195291aca6cb78c5a51aeb410f9b5b4747fdcd32f530f70431e740cc4f50"} Jul 28 23:01:54.081037 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:54.080731 2553 scope.go:117] "RemoveContainer" containerID="31337fcd4b61db326099cef3254156728440f87d92a699354242953547c06acb" Jul 28 23:01:54.091370 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:54.091345 2553 scope.go:117] "RemoveContainer" containerID="31337fcd4b61db326099cef3254156728440f87d92a699354242953547c06acb" Jul 28 23:01:54.091669 ip-10-0-142-132 kubenswrapper[2553]: E0728 23:01:54.091645 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31337fcd4b61db326099cef3254156728440f87d92a699354242953547c06acb\": container with ID starting with 31337fcd4b61db326099cef3254156728440f87d92a699354242953547c06acb not found: ID does not exist" containerID="31337fcd4b61db326099cef3254156728440f87d92a699354242953547c06acb" Jul 28 23:01:54.091780 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:54.091678 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31337fcd4b61db326099cef3254156728440f87d92a699354242953547c06acb"} err="failed to get container status \"31337fcd4b61db326099cef3254156728440f87d92a699354242953547c06acb\": rpc error: code = NotFound desc = could not find container \"31337fcd4b61db326099cef3254156728440f87d92a699354242953547c06acb\": container with ID starting with 31337fcd4b61db326099cef3254156728440f87d92a699354242953547c06acb not found: ID does not exist" Jul 28 23:01:54.102301 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:54.102266 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6c731-559f5fd57b-k6xxd"] Jul 28 23:01:54.104740 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:54.104682 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6c731-559f5fd57b-k6xxd"] Jul 28 23:01:54.406332 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:01:54.406255 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72093d06-b1c0-4bb2-be08-dbf79e6d8d2f" path="/var/lib/kubelet/pods/72093d06-b1c0-4bb2-be08-dbf79e6d8d2f/volumes" Jul 28 23:02:41.572442 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:02:41.572408 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn"] Jul 28 23:02:41.572852 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:02:41.572665 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn" podUID="848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17" containerName="istio-proxy" containerID="cri-o://3ebbb3e99a6fbc340242bae11a276d64e9ac56492e5a6258db1789c7b0aa4929" gracePeriod=30 Jul 28 23:02:46.837692 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:02:46.837623 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn" Jul 28 23:02:46.906081 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:02:46.906044 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17-istio-envoy\") pod \"848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17\" (UID: \"848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17\") " Jul 28 23:02:46.906288 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:02:46.906100 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17-istio-token\") pod \"848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17\" (UID: \"848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17\") " Jul 28 23:02:46.906288 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:02:46.906140 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17-workload-socket\") pod \"848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17\" (UID: \"848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17\") " Jul 28 23:02:46.906288 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:02:46.906168 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17-istiod-ca-cert\") pod \"848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17\" (UID: \"848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17\") " Jul 28 23:02:46.906288 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:02:46.906221 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-btnjt\" (UniqueName: \"kubernetes.io/projected/848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17-kube-api-access-btnjt\") pod \"848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17\" (UID: \"848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17\") " Jul 28 23:02:46.906288 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:02:46.906250 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17-istio-data\") pod \"848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17\" (UID: \"848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17\") " Jul 28 23:02:46.906571 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:02:46.906307 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17-credential-socket\") pod \"848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17\" (UID: \"848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17\") " Jul 28 23:02:46.906571 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:02:46.906367 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17-workload-certs\") pod \"848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17\" (UID: \"848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17\") " Jul 28 23:02:46.906571 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:02:46.906411 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17-istio-podinfo\") pod \"848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17\" (UID: \"848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17\") " Jul 28 23:02:46.906571 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:02:46.906430 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17" (UID: "848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 23:02:46.906834 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:02:46.906692 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17" (UID: "848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 23:02:46.906834 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:02:46.906730 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17" (UID: "848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 23:02:46.906834 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:02:46.906814 2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17-workload-socket\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 23:02:46.906834 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:02:46.906834 2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17-credential-socket\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 23:02:46.907057 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:02:46.906948 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17" (UID: "848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 23:02:46.907966 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:02:46.907934 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17-istio-data" (OuterVolumeSpecName: "istio-data") pod "848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17" (UID: "848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 23:02:46.908568 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:02:46.908548 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17" (UID: "848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 23:02:46.908665 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:02:46.908601 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17-istio-token" (OuterVolumeSpecName: "istio-token") pod "848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17" (UID: "848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 23:02:46.908789 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:02:46.908773 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17-kube-api-access-btnjt" (OuterVolumeSpecName: "kube-api-access-btnjt") pod "848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17" (UID: "848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17"). InnerVolumeSpecName "kube-api-access-btnjt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 23:02:46.909139 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:02:46.909125 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17" (UID: "848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 23:02:47.007301 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:02:47.007252 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btnjt\" (UniqueName: \"kubernetes.io/projected/848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17-kube-api-access-btnjt\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 23:02:47.007301 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:02:47.007293 2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17-istio-data\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 23:02:47.007301 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:02:47.007305 2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17-workload-certs\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 23:02:47.007301 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:02:47.007313 2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17-istio-podinfo\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 23:02:47.007584 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:02:47.007321 2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17-istio-envoy\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 23:02:47.007584 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:02:47.007329 2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17-istio-token\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 23:02:47.007584 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:02:47.007337 2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17-istiod-ca-cert\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 23:02:47.390272 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:02:47.390234 2553 generic.go:356] "Generic (PLEG): container finished" podID="848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17" containerID="3ebbb3e99a6fbc340242bae11a276d64e9ac56492e5a6258db1789c7b0aa4929" exitCode=0 Jul 28 23:02:47.390460 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:02:47.390287 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn" Jul 28 23:02:47.390460 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:02:47.390315 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn" event={"ID":"848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17","Type":"ContainerDied","Data":"3ebbb3e99a6fbc340242bae11a276d64e9ac56492e5a6258db1789c7b0aa4929"} Jul 28 23:02:47.390460 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:02:47.390370 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn" event={"ID":"848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17","Type":"ContainerDied","Data":"c3182e0a83d85d43846ef476c2cac8e62c4f037099894c2a28fffa831fb31909"} Jul 28 23:02:47.390460 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:02:47.390397 2553 scope.go:117] "RemoveContainer" containerID="3ebbb3e99a6fbc340242bae11a276d64e9ac56492e5a6258db1789c7b0aa4929" Jul 28 23:02:47.400815 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:02:47.400796 2553 scope.go:117] "RemoveContainer" containerID="0be8f0e240297bba8fdaad38e2de09fbb8797617cbc31c76537d1cbfc65ff30e" Jul 28 23:02:47.410045 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:02:47.410025 2553 scope.go:117] "RemoveContainer" containerID="3ebbb3e99a6fbc340242bae11a276d64e9ac56492e5a6258db1789c7b0aa4929" Jul 28 23:02:47.410358 ip-10-0-142-132 kubenswrapper[2553]: E0728 23:02:47.410334 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ebbb3e99a6fbc340242bae11a276d64e9ac56492e5a6258db1789c7b0aa4929\": container with ID starting with 3ebbb3e99a6fbc340242bae11a276d64e9ac56492e5a6258db1789c7b0aa4929 not found: ID does not exist" containerID="3ebbb3e99a6fbc340242bae11a276d64e9ac56492e5a6258db1789c7b0aa4929" Jul 28 23:02:47.410424 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:02:47.410367 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ebbb3e99a6fbc340242bae11a276d64e9ac56492e5a6258db1789c7b0aa4929"} err="failed to get container status \"3ebbb3e99a6fbc340242bae11a276d64e9ac56492e5a6258db1789c7b0aa4929\": rpc error: code = NotFound desc = could not find container \"3ebbb3e99a6fbc340242bae11a276d64e9ac56492e5a6258db1789c7b0aa4929\": container with ID starting with 3ebbb3e99a6fbc340242bae11a276d64e9ac56492e5a6258db1789c7b0aa4929 not found: ID does not exist" Jul 28 23:02:47.410424 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:02:47.410387 2553 scope.go:117] "RemoveContainer" containerID="0be8f0e240297bba8fdaad38e2de09fbb8797617cbc31c76537d1cbfc65ff30e" Jul 28 23:02:47.410617 ip-10-0-142-132 kubenswrapper[2553]: E0728 23:02:47.410599 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0be8f0e240297bba8fdaad38e2de09fbb8797617cbc31c76537d1cbfc65ff30e\": container with ID starting with 0be8f0e240297bba8fdaad38e2de09fbb8797617cbc31c76537d1cbfc65ff30e not found: ID does not exist" containerID="0be8f0e240297bba8fdaad38e2de09fbb8797617cbc31c76537d1cbfc65ff30e" Jul 28 23:02:47.410670 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:02:47.410625 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0be8f0e240297bba8fdaad38e2de09fbb8797617cbc31c76537d1cbfc65ff30e"} err="failed to get container status \"0be8f0e240297bba8fdaad38e2de09fbb8797617cbc31c76537d1cbfc65ff30e\": rpc error: code = NotFound desc = could not find container \"0be8f0e240297bba8fdaad38e2de09fbb8797617cbc31c76537d1cbfc65ff30e\": container with ID starting with 0be8f0e240297bba8fdaad38e2de09fbb8797617cbc31c76537d1cbfc65ff30e not found: ID does not exist" Jul 28 23:02:47.412165 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:02:47.412118 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn"] Jul 28 23:02:47.414751 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:02:47.414698 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-c6c731-openshift-default-69fbbf9696-d87kn"] Jul 28 23:02:48.405525 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:02:48.405492 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17" path="/var/lib/kubelet/pods/848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17/volumes" Jul 28 23:02:59.860008 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:02:59.859972 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a134df-5485979d88-p2msq"] Jul 28 23:02:59.860500 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:02:59.860288 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a134df-5485979d88-p2msq" podUID="72cb9aea-6e37-40c7-9a68-af24a8226469" containerName="maas-api" containerID="cri-o://ba650cd157ebc7eafa38ed69fde28aafc2acbb72d929fad21a05f1825d1e146a" gracePeriod=30 Jul 28 23:03:00.126031 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:00.125961 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a134df-5485979d88-p2msq" Jul 28 23:03:00.215051 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:00.215012 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-594h5\" (UniqueName: \"kubernetes.io/projected/72cb9aea-6e37-40c7-9a68-af24a8226469-kube-api-access-594h5\") pod \"72cb9aea-6e37-40c7-9a68-af24a8226469\" (UID: \"72cb9aea-6e37-40c7-9a68-af24a8226469\") " Jul 28 23:03:00.215255 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:00.215100 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72cb9aea-6e37-40c7-9a68-af24a8226469-openshift-service-ca\") pod \"72cb9aea-6e37-40c7-9a68-af24a8226469\" (UID: \"72cb9aea-6e37-40c7-9a68-af24a8226469\") " Jul 28 23:03:00.215255 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:00.215164 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72cb9aea-6e37-40c7-9a68-af24a8226469-maas-api-tls\") pod \"72cb9aea-6e37-40c7-9a68-af24a8226469\" (UID: \"72cb9aea-6e37-40c7-9a68-af24a8226469\") " Jul 28 23:03:00.215626 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:00.215593 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72cb9aea-6e37-40c7-9a68-af24a8226469-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "72cb9aea-6e37-40c7-9a68-af24a8226469" (UID: "72cb9aea-6e37-40c7-9a68-af24a8226469"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 23:03:00.217395 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:00.217374 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72cb9aea-6e37-40c7-9a68-af24a8226469-kube-api-access-594h5" (OuterVolumeSpecName: "kube-api-access-594h5") pod "72cb9aea-6e37-40c7-9a68-af24a8226469" (UID: "72cb9aea-6e37-40c7-9a68-af24a8226469"). InnerVolumeSpecName "kube-api-access-594h5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 23:03:00.217912 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:00.217884 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72cb9aea-6e37-40c7-9a68-af24a8226469-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "72cb9aea-6e37-40c7-9a68-af24a8226469" (UID: "72cb9aea-6e37-40c7-9a68-af24a8226469"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 23:03:00.316415 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:00.316377 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-594h5\" (UniqueName: \"kubernetes.io/projected/72cb9aea-6e37-40c7-9a68-af24a8226469-kube-api-access-594h5\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 23:03:00.316415 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:00.316409 2553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72cb9aea-6e37-40c7-9a68-af24a8226469-openshift-service-ca\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 23:03:00.316415 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:00.316423 2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72cb9aea-6e37-40c7-9a68-af24a8226469-maas-api-tls\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 23:03:00.470113 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:00.470075 2553 generic.go:356] "Generic (PLEG): container finished" podID="72cb9aea-6e37-40c7-9a68-af24a8226469" containerID="ba650cd157ebc7eafa38ed69fde28aafc2acbb72d929fad21a05f1825d1e146a" exitCode=0 Jul 28 23:03:00.470327 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:00.470124 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a134df-5485979d88-p2msq" Jul 28 23:03:00.470327 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:00.470157 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a134df-5485979d88-p2msq" event={"ID":"72cb9aea-6e37-40c7-9a68-af24a8226469","Type":"ContainerDied","Data":"ba650cd157ebc7eafa38ed69fde28aafc2acbb72d929fad21a05f1825d1e146a"} Jul 28 23:03:00.470327 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:00.470212 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a134df-5485979d88-p2msq" event={"ID":"72cb9aea-6e37-40c7-9a68-af24a8226469","Type":"ContainerDied","Data":"43271b505298ea4005de13d4e8f4a46ab597b93b261471f2600dec00b5347e88"} Jul 28 23:03:00.470327 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:00.470235 2553 scope.go:117] "RemoveContainer" containerID="ba650cd157ebc7eafa38ed69fde28aafc2acbb72d929fad21a05f1825d1e146a" Jul 28 23:03:00.479940 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:00.479920 2553 scope.go:117] "RemoveContainer" containerID="ba650cd157ebc7eafa38ed69fde28aafc2acbb72d929fad21a05f1825d1e146a" Jul 28 23:03:00.480226 ip-10-0-142-132 kubenswrapper[2553]: E0728 23:03:00.480203 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba650cd157ebc7eafa38ed69fde28aafc2acbb72d929fad21a05f1825d1e146a\": container with ID starting with ba650cd157ebc7eafa38ed69fde28aafc2acbb72d929fad21a05f1825d1e146a not found: ID does not exist" containerID="ba650cd157ebc7eafa38ed69fde28aafc2acbb72d929fad21a05f1825d1e146a" Jul 28 23:03:00.480279 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:00.480236 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba650cd157ebc7eafa38ed69fde28aafc2acbb72d929fad21a05f1825d1e146a"} err="failed to get container status \"ba650cd157ebc7eafa38ed69fde28aafc2acbb72d929fad21a05f1825d1e146a\": rpc error: code = NotFound desc = could not find container \"ba650cd157ebc7eafa38ed69fde28aafc2acbb72d929fad21a05f1825d1e146a\": container with ID starting with ba650cd157ebc7eafa38ed69fde28aafc2acbb72d929fad21a05f1825d1e146a not found: ID does not exist" Jul 28 23:03:00.484743 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:00.484720 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a134df-5485979d88-p2msq"] Jul 28 23:03:00.489993 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:00.489970 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a134df-5485979d88-p2msq"] Jul 28 23:03:02.404951 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:02.404919 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72cb9aea-6e37-40c7-9a68-af24a8226469" path="/var/lib/kubelet/pods/72cb9aea-6e37-40c7-9a68-af24a8226469/volumes" Jul 28 23:03:06.504431 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:06.504393 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh"] Jul 28 23:03:06.504995 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:06.504628 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh" podUID="c04fb80a-7dab-4e27-84c3-687d81d5ce99" containerName="istio-proxy" containerID="cri-o://c5b6ebb9d07e9fa861690f1b7e6e777b7266f3730f5afb1d95c4850c1c54350f" gracePeriod=30 Jul 28 23:03:11.763104 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:11.763076 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh" Jul 28 23:03:11.825388 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:11.825309 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c04fb80a-7dab-4e27-84c3-687d81d5ce99-istiod-ca-cert\") pod \"c04fb80a-7dab-4e27-84c3-687d81d5ce99\" (UID: \"c04fb80a-7dab-4e27-84c3-687d81d5ce99\") " Jul 28 23:03:11.825388 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:11.825344 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w7f2v\" (UniqueName: \"kubernetes.io/projected/c04fb80a-7dab-4e27-84c3-687d81d5ce99-kube-api-access-w7f2v\") pod \"c04fb80a-7dab-4e27-84c3-687d81d5ce99\" (UID: \"c04fb80a-7dab-4e27-84c3-687d81d5ce99\") " Jul 28 23:03:11.825388 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:11.825372 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c04fb80a-7dab-4e27-84c3-687d81d5ce99-istio-envoy\") pod \"c04fb80a-7dab-4e27-84c3-687d81d5ce99\" (UID: \"c04fb80a-7dab-4e27-84c3-687d81d5ce99\") " Jul 28 23:03:11.825604 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:11.825438 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c04fb80a-7dab-4e27-84c3-687d81d5ce99-istio-data\") pod \"c04fb80a-7dab-4e27-84c3-687d81d5ce99\" (UID: \"c04fb80a-7dab-4e27-84c3-687d81d5ce99\") " Jul 28 23:03:11.825604 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:11.825473 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04fb80a-7dab-4e27-84c3-687d81d5ce99-credential-socket\") pod \"c04fb80a-7dab-4e27-84c3-687d81d5ce99\" (UID: \"c04fb80a-7dab-4e27-84c3-687d81d5ce99\") " Jul 28 23:03:11.825604 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:11.825513 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04fb80a-7dab-4e27-84c3-687d81d5ce99-workload-socket\") pod \"c04fb80a-7dab-4e27-84c3-687d81d5ce99\" (UID: \"c04fb80a-7dab-4e27-84c3-687d81d5ce99\") " Jul 28 23:03:11.825604 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:11.825537 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c04fb80a-7dab-4e27-84c3-687d81d5ce99-istio-token\") pod \"c04fb80a-7dab-4e27-84c3-687d81d5ce99\" (UID: \"c04fb80a-7dab-4e27-84c3-687d81d5ce99\") " Jul 28 23:03:11.825604 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:11.825559 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c04fb80a-7dab-4e27-84c3-687d81d5ce99-istio-podinfo\") pod \"c04fb80a-7dab-4e27-84c3-687d81d5ce99\" (UID: \"c04fb80a-7dab-4e27-84c3-687d81d5ce99\") " Jul 28 23:03:11.825899 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:11.825614 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c04fb80a-7dab-4e27-84c3-687d81d5ce99-workload-certs\") pod \"c04fb80a-7dab-4e27-84c3-687d81d5ce99\" (UID: \"c04fb80a-7dab-4e27-84c3-687d81d5ce99\") " Jul 28 23:03:11.825899 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:11.825725 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c04fb80a-7dab-4e27-84c3-687d81d5ce99-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c04fb80a-7dab-4e27-84c3-687d81d5ce99" (UID: "c04fb80a-7dab-4e27-84c3-687d81d5ce99"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 23:03:11.825899 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:11.825818 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c04fb80a-7dab-4e27-84c3-687d81d5ce99-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c04fb80a-7dab-4e27-84c3-687d81d5ce99" (UID: "c04fb80a-7dab-4e27-84c3-687d81d5ce99"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 23:03:11.825899 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:11.825831 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c04fb80a-7dab-4e27-84c3-687d81d5ce99-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c04fb80a-7dab-4e27-84c3-687d81d5ce99" (UID: "c04fb80a-7dab-4e27-84c3-687d81d5ce99"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 23:03:11.826085 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:11.825983 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c04fb80a-7dab-4e27-84c3-687d81d5ce99-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c04fb80a-7dab-4e27-84c3-687d81d5ce99" (UID: "c04fb80a-7dab-4e27-84c3-687d81d5ce99"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 23:03:11.826085 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:11.825994 2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c04fb80a-7dab-4e27-84c3-687d81d5ce99-istiod-ca-cert\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 23:03:11.826085 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:11.826020 2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04fb80a-7dab-4e27-84c3-687d81d5ce99-credential-socket\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 23:03:11.826085 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:11.826036 2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04fb80a-7dab-4e27-84c3-687d81d5ce99-workload-socket\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 23:03:11.826973 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:11.826936 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c04fb80a-7dab-4e27-84c3-687d81d5ce99-istio-data" (OuterVolumeSpecName: "istio-data") pod "c04fb80a-7dab-4e27-84c3-687d81d5ce99" (UID: "c04fb80a-7dab-4e27-84c3-687d81d5ce99"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 23:03:11.827963 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:11.827931 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c04fb80a-7dab-4e27-84c3-687d81d5ce99-kube-api-access-w7f2v" (OuterVolumeSpecName: "kube-api-access-w7f2v") pod "c04fb80a-7dab-4e27-84c3-687d81d5ce99" (UID: "c04fb80a-7dab-4e27-84c3-687d81d5ce99"). InnerVolumeSpecName "kube-api-access-w7f2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 23:03:11.827963 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:11.827936 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c04fb80a-7dab-4e27-84c3-687d81d5ce99-istio-token" (OuterVolumeSpecName: "istio-token") pod "c04fb80a-7dab-4e27-84c3-687d81d5ce99" (UID: "c04fb80a-7dab-4e27-84c3-687d81d5ce99"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 23:03:11.828090 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:11.828019 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c04fb80a-7dab-4e27-84c3-687d81d5ce99-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c04fb80a-7dab-4e27-84c3-687d81d5ce99" (UID: "c04fb80a-7dab-4e27-84c3-687d81d5ce99"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 23:03:11.828445 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:11.828419 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c04fb80a-7dab-4e27-84c3-687d81d5ce99-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c04fb80a-7dab-4e27-84c3-687d81d5ce99" (UID: "c04fb80a-7dab-4e27-84c3-687d81d5ce99"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 23:03:11.927247 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:11.927217 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7f2v\" (UniqueName: \"kubernetes.io/projected/c04fb80a-7dab-4e27-84c3-687d81d5ce99-kube-api-access-w7f2v\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 23:03:11.927247 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:11.927243 2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c04fb80a-7dab-4e27-84c3-687d81d5ce99-istio-envoy\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 23:03:11.927247 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:11.927252 2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c04fb80a-7dab-4e27-84c3-687d81d5ce99-istio-data\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 23:03:11.927530 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:11.927260 2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c04fb80a-7dab-4e27-84c3-687d81d5ce99-istio-token\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 23:03:11.927530 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:11.927267 2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c04fb80a-7dab-4e27-84c3-687d81d5ce99-istio-podinfo\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 23:03:11.927530 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:11.927276 2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c04fb80a-7dab-4e27-84c3-687d81d5ce99-workload-certs\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 23:03:12.537800 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:12.537764 2553 generic.go:356] "Generic (PLEG): container finished" podID="c04fb80a-7dab-4e27-84c3-687d81d5ce99" containerID="c5b6ebb9d07e9fa861690f1b7e6e777b7266f3730f5afb1d95c4850c1c54350f" exitCode=0 Jul 28 23:03:12.538004 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:12.537807 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh" Jul 28 23:03:12.538004 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:12.537847 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh" event={"ID":"c04fb80a-7dab-4e27-84c3-687d81d5ce99","Type":"ContainerDied","Data":"c5b6ebb9d07e9fa861690f1b7e6e777b7266f3730f5afb1d95c4850c1c54350f"} Jul 28 23:03:12.538004 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:12.537905 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh" event={"ID":"c04fb80a-7dab-4e27-84c3-687d81d5ce99","Type":"ContainerDied","Data":"7799a9fb7010a9d9540534fd53470adeaa03779bc4174f7ec8a8a7ebe80f0e2c"} Jul 28 23:03:12.538004 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:12.537935 2553 scope.go:117] "RemoveContainer" containerID="c5b6ebb9d07e9fa861690f1b7e6e777b7266f3730f5afb1d95c4850c1c54350f" Jul 28 23:03:12.547104 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:12.547085 2553 scope.go:117] "RemoveContainer" containerID="848dca6aef0c8293024aa657d213ec40f085bde8d09fd00f6b430cf521bbf347" Jul 28 23:03:12.555594 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:12.555566 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh"] Jul 28 23:03:12.557521 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:12.557505 2553 scope.go:117] "RemoveContainer" containerID="c5b6ebb9d07e9fa861690f1b7e6e777b7266f3730f5afb1d95c4850c1c54350f" Jul 28 23:03:12.557854 ip-10-0-142-132 kubenswrapper[2553]: E0728 23:03:12.557831 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5b6ebb9d07e9fa861690f1b7e6e777b7266f3730f5afb1d95c4850c1c54350f\": container with ID starting with c5b6ebb9d07e9fa861690f1b7e6e777b7266f3730f5afb1d95c4850c1c54350f not found: ID does not exist" containerID="c5b6ebb9d07e9fa861690f1b7e6e777b7266f3730f5afb1d95c4850c1c54350f" Jul 28 23:03:12.557945 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:12.557863 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5b6ebb9d07e9fa861690f1b7e6e777b7266f3730f5afb1d95c4850c1c54350f"} err="failed to get container status \"c5b6ebb9d07e9fa861690f1b7e6e777b7266f3730f5afb1d95c4850c1c54350f\": rpc error: code = NotFound desc = could not find container \"c5b6ebb9d07e9fa861690f1b7e6e777b7266f3730f5afb1d95c4850c1c54350f\": container with ID starting with c5b6ebb9d07e9fa861690f1b7e6e777b7266f3730f5afb1d95c4850c1c54350f not found: ID does not exist" Jul 28 23:03:12.557945 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:12.557882 2553 scope.go:117] "RemoveContainer" containerID="848dca6aef0c8293024aa657d213ec40f085bde8d09fd00f6b430cf521bbf347" Jul 28 23:03:12.558155 ip-10-0-142-132 kubenswrapper[2553]: E0728 23:03:12.558135 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"848dca6aef0c8293024aa657d213ec40f085bde8d09fd00f6b430cf521bbf347\": container with ID starting with 848dca6aef0c8293024aa657d213ec40f085bde8d09fd00f6b430cf521bbf347 not found: ID does not exist" containerID="848dca6aef0c8293024aa657d213ec40f085bde8d09fd00f6b430cf521bbf347" Jul 28 23:03:12.558215 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:12.558162 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"848dca6aef0c8293024aa657d213ec40f085bde8d09fd00f6b430cf521bbf347"} err="failed to get container status \"848dca6aef0c8293024aa657d213ec40f085bde8d09fd00f6b430cf521bbf347\": rpc error: code = NotFound desc = could not find container \"848dca6aef0c8293024aa657d213ec40f085bde8d09fd00f6b430cf521bbf347\": container with ID starting with 848dca6aef0c8293024aa657d213ec40f085bde8d09fd00f6b430cf521bbf347 not found: ID does not exist" Jul 28 23:03:12.559504 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:12.559480 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-a134df-openshift-default-68cf665b6d-8z5sh"] Jul 28 23:03:14.404883 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:14.404852 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c04fb80a-7dab-4e27-84c3-687d81d5ce99" path="/var/lib/kubelet/pods/c04fb80a-7dab-4e27-84c3-687d81d5ce99/volumes" Jul 28 23:03:23.899182 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:23.899147 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4"] Jul 28 23:03:23.899686 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:23.899380 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4" podUID="cdd68e6b-b347-4c96-b476-f3341089ff50" containerName="istio-proxy" containerID="cri-o://80d4820ac8d4583e45bd69cba29aa2b9495f142ca7e30bcdc190f2fc9f679f30" gracePeriod=30 Jul 28 23:03:29.155818 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:29.155790 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4" Jul 28 23:03:29.285088 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:29.285051 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdd68e6b-b347-4c96-b476-f3341089ff50-workload-socket\") pod \"cdd68e6b-b347-4c96-b476-f3341089ff50\" (UID: \"cdd68e6b-b347-4c96-b476-f3341089ff50\") " Jul 28 23:03:29.285088 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:29.285088 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdd68e6b-b347-4c96-b476-f3341089ff50-istio-podinfo\") pod \"cdd68e6b-b347-4c96-b476-f3341089ff50\" (UID: \"cdd68e6b-b347-4c96-b476-f3341089ff50\") " Jul 28 23:03:29.285344 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:29.285123 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdd68e6b-b347-4c96-b476-f3341089ff50-istiod-ca-cert\") pod \"cdd68e6b-b347-4c96-b476-f3341089ff50\" (UID: \"cdd68e6b-b347-4c96-b476-f3341089ff50\") " Jul 28 23:03:29.285344 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:29.285155 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t59g6\" (UniqueName: \"kubernetes.io/projected/cdd68e6b-b347-4c96-b476-f3341089ff50-kube-api-access-t59g6\") pod \"cdd68e6b-b347-4c96-b476-f3341089ff50\" (UID: \"cdd68e6b-b347-4c96-b476-f3341089ff50\") " Jul 28 23:03:29.285344 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:29.285261 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdd68e6b-b347-4c96-b476-f3341089ff50-credential-socket\") pod \"cdd68e6b-b347-4c96-b476-f3341089ff50\" (UID: \"cdd68e6b-b347-4c96-b476-f3341089ff50\") " Jul 28 23:03:29.285344 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:29.285307 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdd68e6b-b347-4c96-b476-f3341089ff50-workload-certs\") pod \"cdd68e6b-b347-4c96-b476-f3341089ff50\" (UID: \"cdd68e6b-b347-4c96-b476-f3341089ff50\") " Jul 28 23:03:29.285344 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:29.285336 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdd68e6b-b347-4c96-b476-f3341089ff50-istio-token\") pod \"cdd68e6b-b347-4c96-b476-f3341089ff50\" (UID: \"cdd68e6b-b347-4c96-b476-f3341089ff50\") " Jul 28 23:03:29.285619 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:29.285386 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdd68e6b-b347-4c96-b476-f3341089ff50-istio-data\") pod \"cdd68e6b-b347-4c96-b476-f3341089ff50\" (UID: \"cdd68e6b-b347-4c96-b476-f3341089ff50\") " Jul 28 23:03:29.285619 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:29.285412 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdd68e6b-b347-4c96-b476-f3341089ff50-istio-envoy\") pod \"cdd68e6b-b347-4c96-b476-f3341089ff50\" (UID: \"cdd68e6b-b347-4c96-b476-f3341089ff50\") " Jul 28 23:03:29.285619 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:29.285411 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdd68e6b-b347-4c96-b476-f3341089ff50-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cdd68e6b-b347-4c96-b476-f3341089ff50" (UID: "cdd68e6b-b347-4c96-b476-f3341089ff50"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 23:03:29.285619 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:29.285577 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdd68e6b-b347-4c96-b476-f3341089ff50-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cdd68e6b-b347-4c96-b476-f3341089ff50" (UID: "cdd68e6b-b347-4c96-b476-f3341089ff50"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 23:03:29.285875 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:29.285657 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdd68e6b-b347-4c96-b476-f3341089ff50-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cdd68e6b-b347-4c96-b476-f3341089ff50" (UID: "cdd68e6b-b347-4c96-b476-f3341089ff50"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 23:03:29.285875 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:29.285586 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdd68e6b-b347-4c96-b476-f3341089ff50-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cdd68e6b-b347-4c96-b476-f3341089ff50" (UID: "cdd68e6b-b347-4c96-b476-f3341089ff50"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 23:03:29.285875 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:29.285823 2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdd68e6b-b347-4c96-b476-f3341089ff50-workload-socket\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 23:03:29.285875 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:29.285844 2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdd68e6b-b347-4c96-b476-f3341089ff50-istiod-ca-cert\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 23:03:29.285875 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:29.285856 2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdd68e6b-b347-4c96-b476-f3341089ff50-credential-socket\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 23:03:29.285875 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:29.285869 2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdd68e6b-b347-4c96-b476-f3341089ff50-workload-certs\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 23:03:29.286390 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:29.286368 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdd68e6b-b347-4c96-b476-f3341089ff50-istio-data" (OuterVolumeSpecName: "istio-data") pod "cdd68e6b-b347-4c96-b476-f3341089ff50" (UID: "cdd68e6b-b347-4c96-b476-f3341089ff50"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 23:03:29.287613 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:29.287594 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cdd68e6b-b347-4c96-b476-f3341089ff50-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cdd68e6b-b347-4c96-b476-f3341089ff50" (UID: "cdd68e6b-b347-4c96-b476-f3341089ff50"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 23:03:29.287806 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:29.287690 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdd68e6b-b347-4c96-b476-f3341089ff50-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cdd68e6b-b347-4c96-b476-f3341089ff50" (UID: "cdd68e6b-b347-4c96-b476-f3341089ff50"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 23:03:29.288041 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:29.288021 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdd68e6b-b347-4c96-b476-f3341089ff50-kube-api-access-t59g6" (OuterVolumeSpecName: "kube-api-access-t59g6") pod "cdd68e6b-b347-4c96-b476-f3341089ff50" (UID: "cdd68e6b-b347-4c96-b476-f3341089ff50"). InnerVolumeSpecName "kube-api-access-t59g6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 23:03:29.288124 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:29.288103 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdd68e6b-b347-4c96-b476-f3341089ff50-istio-token" (OuterVolumeSpecName: "istio-token") pod "cdd68e6b-b347-4c96-b476-f3341089ff50" (UID: "cdd68e6b-b347-4c96-b476-f3341089ff50"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 23:03:29.386663 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:29.386630 2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdd68e6b-b347-4c96-b476-f3341089ff50-istio-podinfo\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 23:03:29.386663 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:29.386654 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t59g6\" (UniqueName: \"kubernetes.io/projected/cdd68e6b-b347-4c96-b476-f3341089ff50-kube-api-access-t59g6\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 23:03:29.386663 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:29.386664 2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdd68e6b-b347-4c96-b476-f3341089ff50-istio-token\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 23:03:29.386663 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:29.386672 2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdd68e6b-b347-4c96-b476-f3341089ff50-istio-data\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 23:03:29.386930 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:29.386679 2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdd68e6b-b347-4c96-b476-f3341089ff50-istio-envoy\") on node \"ip-10-0-142-132.ec2.internal\" DevicePath \"\"" Jul 28 23:03:29.633588 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:29.633497 2553 generic.go:356] "Generic (PLEG): container finished" podID="cdd68e6b-b347-4c96-b476-f3341089ff50" containerID="80d4820ac8d4583e45bd69cba29aa2b9495f142ca7e30bcdc190f2fc9f679f30" exitCode=0 Jul 28 23:03:29.633588 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:29.633556 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4" Jul 28 23:03:29.633797 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:29.633584 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4" event={"ID":"cdd68e6b-b347-4c96-b476-f3341089ff50","Type":"ContainerDied","Data":"80d4820ac8d4583e45bd69cba29aa2b9495f142ca7e30bcdc190f2fc9f679f30"} Jul 28 23:03:29.633797 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:29.633656 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4" event={"ID":"cdd68e6b-b347-4c96-b476-f3341089ff50","Type":"ContainerDied","Data":"75b271c2e0e6f1c4b97a761a4ab134ede2a8c3976bd6da9d61451fa964da4405"} Jul 28 23:03:29.633797 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:29.633679 2553 scope.go:117] "RemoveContainer" containerID="80d4820ac8d4583e45bd69cba29aa2b9495f142ca7e30bcdc190f2fc9f679f30" Jul 28 23:03:29.643413 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:29.643397 2553 scope.go:117] "RemoveContainer" containerID="72b9a1d2a5f85a23efa8de64794dd6071ebb0f4858c755991a6096f945b97850" Jul 28 23:03:29.652022 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:29.651999 2553 scope.go:117] "RemoveContainer" containerID="80d4820ac8d4583e45bd69cba29aa2b9495f142ca7e30bcdc190f2fc9f679f30" Jul 28 23:03:29.652405 ip-10-0-142-132 kubenswrapper[2553]: E0728 23:03:29.652377 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80d4820ac8d4583e45bd69cba29aa2b9495f142ca7e30bcdc190f2fc9f679f30\": container with ID starting with 80d4820ac8d4583e45bd69cba29aa2b9495f142ca7e30bcdc190f2fc9f679f30 not found: ID does not exist" containerID="80d4820ac8d4583e45bd69cba29aa2b9495f142ca7e30bcdc190f2fc9f679f30" Jul 28 23:03:29.652503 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:29.652415 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80d4820ac8d4583e45bd69cba29aa2b9495f142ca7e30bcdc190f2fc9f679f30"} err="failed to get container status \"80d4820ac8d4583e45bd69cba29aa2b9495f142ca7e30bcdc190f2fc9f679f30\": rpc error: code = NotFound desc = could not find container \"80d4820ac8d4583e45bd69cba29aa2b9495f142ca7e30bcdc190f2fc9f679f30\": container with ID starting with 80d4820ac8d4583e45bd69cba29aa2b9495f142ca7e30bcdc190f2fc9f679f30 not found: ID does not exist" Jul 28 23:03:29.652503 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:29.652438 2553 scope.go:117] "RemoveContainer" containerID="72b9a1d2a5f85a23efa8de64794dd6071ebb0f4858c755991a6096f945b97850" Jul 28 23:03:29.652697 ip-10-0-142-132 kubenswrapper[2553]: E0728 23:03:29.652681 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72b9a1d2a5f85a23efa8de64794dd6071ebb0f4858c755991a6096f945b97850\": container with ID starting with 72b9a1d2a5f85a23efa8de64794dd6071ebb0f4858c755991a6096f945b97850 not found: ID does not exist" containerID="72b9a1d2a5f85a23efa8de64794dd6071ebb0f4858c755991a6096f945b97850" Jul 28 23:03:29.652767 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:29.652728 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72b9a1d2a5f85a23efa8de64794dd6071ebb0f4858c755991a6096f945b97850"} err="failed to get container status \"72b9a1d2a5f85a23efa8de64794dd6071ebb0f4858c755991a6096f945b97850\": rpc error: code = NotFound desc = could not find container \"72b9a1d2a5f85a23efa8de64794dd6071ebb0f4858c755991a6096f945b97850\": container with ID starting with 72b9a1d2a5f85a23efa8de64794dd6071ebb0f4858c755991a6096f945b97850 not found: ID does not exist" Jul 28 23:03:29.654343 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:29.654324 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4"] Jul 28 23:03:29.658249 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:29.658229 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-faf483-openshift-default-79c444b5b8-vqsw4"] Jul 28 23:03:30.170993 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:30.170969 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-d6457fd9-dghbg_4184e9a3-2f74-4275-8c85-5764766290cd/manager/0.log" Jul 28 23:03:30.405690 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:30.405656 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cdd68e6b-b347-4c96-b476-f3341089ff50" path="/var/lib/kubelet/pods/cdd68e6b-b347-4c96-b476-f3341089ff50/volumes" Jul 28 23:03:30.644668 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:30.644636 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b7ccc5769-88sx5_331d730d-2853-4509-9e90-4727ad3ed40d/copy-manifests/0.log" Jul 28 23:03:30.650862 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:30.650840 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b7ccc5769-88sx5_331d730d-2853-4509-9e90-4727ad3ed40d/manager/0.log" Jul 28 23:03:31.127566 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:31.127536 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5f4497d45c-qdp9t_49752556-bd0b-46ea-b3fd-d4b807db64c7/maas-api/0.log" Jul 28 23:03:32.323247 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:32.323220 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hm82z_a2c4df61-0316-4955-97dc-84b857a09e45/util/0.log" Jul 28 23:03:32.329062 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:32.329019 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hm82z_a2c4df61-0316-4955-97dc-84b857a09e45/pull/0.log" Jul 28 23:03:32.334287 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:32.334263 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hm82z_a2c4df61-0316-4955-97dc-84b857a09e45/extract/0.log" Jul 28 23:03:32.443183 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:32.443147 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sbp9r_6f8736b6-aec3-4649-9ea2-a7425f6cc890/util/0.log" Jul 28 23:03:32.449071 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:32.449040 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sbp9r_6f8736b6-aec3-4649-9ea2-a7425f6cc890/pull/0.log" Jul 28 23:03:32.454633 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:32.454615 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sbp9r_6f8736b6-aec3-4649-9ea2-a7425f6cc890/extract/0.log" Jul 28 23:03:32.565124 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:32.565089 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p4rp6_9ecff602-ff0f-4275-92e1-24ed7c903af2/pull/0.log" Jul 28 23:03:32.572924 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:32.572899 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p4rp6_9ecff602-ff0f-4275-92e1-24ed7c903af2/extract/0.log" Jul 28 23:03:32.579797 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:32.579727 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p4rp6_9ecff602-ff0f-4275-92e1-24ed7c903af2/util/0.log" Jul 28 23:03:32.695258 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:32.695224 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1frrnl_39ea32e0-a0d7-434f-832d-3a934734b409/util/0.log" Jul 28 23:03:32.701420 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:32.701396 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1frrnl_39ea32e0-a0d7-434f-832d-3a934734b409/pull/0.log" Jul 28 23:03:32.707832 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:32.707803 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1frrnl_39ea32e0-a0d7-434f-832d-3a934734b409/extract/0.log" Jul 28 23:03:34.129073 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:34.129037 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-vdkjj_c283e4ed-82e2-474a-b100-c34afa239b17/discovery/0.log" Jul 28 23:03:34.479363 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:34.479335 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-bfrsf_e776e265-c83c-42be-be8c-07647513e2ba/istio-proxy/1.log" Jul 28 23:03:34.845212 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:34.839419 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6747b7576c-fs2bd_7e266916-99eb-4807-9b85-fabdeb9e1c28/router/0.log" Jul 28 23:03:35.191067 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:35.190986 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-7rsjb_75574aba-fe8f-451e-95f1-d11be8d7249a/main/0.log" Jul 28 23:03:35.530051 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:35.530024 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbwgdp_d7adda96-3dfd-42f9-987e-2aeed5d39c49/main/0.log" Jul 28 23:03:44.216537 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:44.216498 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-djzl8_59a1f798-88a8-4236-82f2-0a675d85920d/global-pull-secret-syncer/0.log" Jul 28 23:03:44.317955 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:44.317914 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-7q942_c93df2e7-d79a-4a98-8556-a98868dfd93e/konnectivity-agent/0.log" Jul 28 23:03:44.427659 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:44.427628 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-132.ec2.internal_84e2d03b22c44b29314bad0b0a3a050e/haproxy/0.log" Jul 28 23:03:49.392261 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:49.392231 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hm82z_a2c4df61-0316-4955-97dc-84b857a09e45/extract/0.log" Jul 28 23:03:49.421155 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:49.421123 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hm82z_a2c4df61-0316-4955-97dc-84b857a09e45/util/0.log" Jul 28 23:03:49.443305 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:49.443275 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hm82z_a2c4df61-0316-4955-97dc-84b857a09e45/pull/0.log" Jul 28 23:03:49.471897 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:49.471862 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sbp9r_6f8736b6-aec3-4649-9ea2-a7425f6cc890/extract/0.log" Jul 28 23:03:49.494205 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:49.494181 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sbp9r_6f8736b6-aec3-4649-9ea2-a7425f6cc890/util/0.log" Jul 28 23:03:49.515484 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:49.515461 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sbp9r_6f8736b6-aec3-4649-9ea2-a7425f6cc890/pull/0.log" Jul 28 23:03:49.541241 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:49.541211 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p4rp6_9ecff602-ff0f-4275-92e1-24ed7c903af2/extract/0.log" Jul 28 23:03:49.564880 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:49.564856 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p4rp6_9ecff602-ff0f-4275-92e1-24ed7c903af2/util/0.log" Jul 28 23:03:49.586222 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:49.586204 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p4rp6_9ecff602-ff0f-4275-92e1-24ed7c903af2/pull/0.log" Jul 28 23:03:49.611854 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:49.611833 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1frrnl_39ea32e0-a0d7-434f-832d-3a934734b409/extract/0.log" Jul 28 23:03:49.633199 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:49.633177 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1frrnl_39ea32e0-a0d7-434f-832d-3a934734b409/util/0.log" Jul 28 23:03:49.655325 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:49.655260 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1frrnl_39ea32e0-a0d7-434f-832d-3a934734b409/pull/0.log" Jul 28 23:03:52.603218 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:52.603182 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-zn5fm_139cae0a-1747-4e84-b210-24cac8e94541/cluster-monitoring-operator/0.log" Jul 28 23:03:52.930631 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:52.930550 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vnzs2_b7760431-5b30-439d-83dc-f5fdaec65569/node-exporter/0.log" Jul 28 23:03:52.959331 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:52.959306 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vnzs2_b7760431-5b30-439d-83dc-f5fdaec65569/kube-rbac-proxy/0.log" Jul 28 23:03:52.984713 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:52.984690 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vnzs2_b7760431-5b30-439d-83dc-f5fdaec65569/init-textfile/0.log" Jul 28 23:03:54.169731 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:54.169678 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tzj6t/perf-node-gather-daemonset-45xgp"] Jul 28 23:03:54.170207 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:54.170183 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17" containerName="istio-proxy" Jul 28 23:03:54.170354 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:54.170334 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17" containerName="istio-proxy" Jul 28 23:03:54.170435 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:54.170361 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bb95084-bc1c-4a63-87c2-8f9594d8fb4a" containerName="istio-proxy" Jul 28 23:03:54.170435 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:54.170371 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bb95084-bc1c-4a63-87c2-8f9594d8fb4a" containerName="istio-proxy" Jul 28 23:03:54.170435 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:54.170390 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e5ecb88-9e83-4f8f-be96-bc5570d3cdcd" containerName="maas-api" Jul 28 23:03:54.170435 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:54.170399 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e5ecb88-9e83-4f8f-be96-bc5570d3cdcd" containerName="maas-api" Jul 28 23:03:54.170435 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:54.170416 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72093d06-b1c0-4bb2-be08-dbf79e6d8d2f" containerName="maas-api" Jul 28 23:03:54.170435 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:54.170424 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="72093d06-b1c0-4bb2-be08-dbf79e6d8d2f" containerName="maas-api" Jul 28 23:03:54.170777 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:54.170450 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72cb9aea-6e37-40c7-9a68-af24a8226469" containerName="maas-api" Jul 28 23:03:54.170777 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:54.170458 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="72cb9aea-6e37-40c7-9a68-af24a8226469" containerName="maas-api" Jul 28 23:03:54.170777 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:54.170470 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c04fb80a-7dab-4e27-84c3-687d81d5ce99" containerName="istio-proxy" Jul 28 23:03:54.170777 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:54.170478 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="c04fb80a-7dab-4e27-84c3-687d81d5ce99" containerName="istio-proxy" Jul 28 23:03:54.170777 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:54.170487 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdd68e6b-b347-4c96-b476-f3341089ff50" containerName="istio-proxy" Jul 28 23:03:54.170777 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:54.170496 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdd68e6b-b347-4c96-b476-f3341089ff50" containerName="istio-proxy" Jul 28 23:03:54.170777 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:54.170509 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdd68e6b-b347-4c96-b476-f3341089ff50" containerName="istio-proxy" Jul 28 23:03:54.170777 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:54.170518 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdd68e6b-b347-4c96-b476-f3341089ff50" containerName="istio-proxy" Jul 28 23:03:54.170777 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:54.170532 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c04fb80a-7dab-4e27-84c3-687d81d5ce99" containerName="istio-proxy" Jul 28 23:03:54.170777 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:54.170541 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="c04fb80a-7dab-4e27-84c3-687d81d5ce99" containerName="istio-proxy" Jul 28 23:03:54.170777 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:54.170636 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="c04fb80a-7dab-4e27-84c3-687d81d5ce99" containerName="istio-proxy" Jul 28 23:03:54.170777 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:54.170647 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdd68e6b-b347-4c96-b476-f3341089ff50" containerName="istio-proxy" Jul 28 23:03:54.170777 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:54.170659 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdd68e6b-b347-4c96-b476-f3341089ff50" containerName="istio-proxy" Jul 28 23:03:54.170777 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:54.170668 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e5ecb88-9e83-4f8f-be96-bc5570d3cdcd" containerName="maas-api" Jul 28 23:03:54.170777 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:54.170678 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="72093d06-b1c0-4bb2-be08-dbf79e6d8d2f" containerName="maas-api" Jul 28 23:03:54.170777 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:54.170688 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bb95084-bc1c-4a63-87c2-8f9594d8fb4a" containerName="istio-proxy" Jul 28 23:03:54.170777 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:54.170695 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17" containerName="istio-proxy" Jul 28 23:03:54.170777 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:54.170725 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="72cb9aea-6e37-40c7-9a68-af24a8226469" containerName="maas-api" Jul 28 23:03:54.170777 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:54.170732 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="848b7fe8-0ad6-49ea-9fb8-1ce2d2e46f17" containerName="istio-proxy" Jul 28 23:03:54.170777 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:54.170741 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bb95084-bc1c-4a63-87c2-8f9594d8fb4a" containerName="istio-proxy" Jul 28 23:03:54.174171 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:54.174150 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tzj6t/perf-node-gather-daemonset-45xgp" Jul 28 23:03:54.176089 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:54.176061 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tzj6t\"/\"kube-root-ca.crt\"" Jul 28 23:03:54.176227 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:54.176206 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tzj6t\"/\"openshift-service-ca.crt\"" Jul 28 23:03:54.177023 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:54.177004 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-tzj6t\"/\"default-dockercfg-gslgj\"" Jul 28 23:03:54.182882 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:54.182859 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tzj6t/perf-node-gather-daemonset-45xgp"] Jul 28 23:03:54.216184 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:54.216141 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4d1bfed9-2800-4762-a98c-16f450c58bd3-lib-modules\") pod \"perf-node-gather-daemonset-45xgp\" (UID: \"4d1bfed9-2800-4762-a98c-16f450c58bd3\") " pod="openshift-must-gather-tzj6t/perf-node-gather-daemonset-45xgp" Jul 28 23:03:54.216184 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:54.216185 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4d1bfed9-2800-4762-a98c-16f450c58bd3-sys\") pod \"perf-node-gather-daemonset-45xgp\" (UID: \"4d1bfed9-2800-4762-a98c-16f450c58bd3\") " pod="openshift-must-gather-tzj6t/perf-node-gather-daemonset-45xgp" Jul 28 23:03:54.216411 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:54.216227 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4d1bfed9-2800-4762-a98c-16f450c58bd3-podres\") pod \"perf-node-gather-daemonset-45xgp\" (UID: \"4d1bfed9-2800-4762-a98c-16f450c58bd3\") " pod="openshift-must-gather-tzj6t/perf-node-gather-daemonset-45xgp" Jul 28 23:03:54.216411 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:54.216267 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmxq2\" (UniqueName: \"kubernetes.io/projected/4d1bfed9-2800-4762-a98c-16f450c58bd3-kube-api-access-mmxq2\") pod \"perf-node-gather-daemonset-45xgp\" (UID: \"4d1bfed9-2800-4762-a98c-16f450c58bd3\") " pod="openshift-must-gather-tzj6t/perf-node-gather-daemonset-45xgp" Jul 28 23:03:54.216411 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:54.216302 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4d1bfed9-2800-4762-a98c-16f450c58bd3-proc\") pod \"perf-node-gather-daemonset-45xgp\" (UID: \"4d1bfed9-2800-4762-a98c-16f450c58bd3\") " pod="openshift-must-gather-tzj6t/perf-node-gather-daemonset-45xgp" Jul 28 23:03:54.317379 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:54.317340 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4d1bfed9-2800-4762-a98c-16f450c58bd3-proc\") pod \"perf-node-gather-daemonset-45xgp\" (UID: \"4d1bfed9-2800-4762-a98c-16f450c58bd3\") " pod="openshift-must-gather-tzj6t/perf-node-gather-daemonset-45xgp" Jul 28 23:03:54.317560 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:54.317407 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4d1bfed9-2800-4762-a98c-16f450c58bd3-lib-modules\") pod \"perf-node-gather-daemonset-45xgp\" (UID: \"4d1bfed9-2800-4762-a98c-16f450c58bd3\") " pod="openshift-must-gather-tzj6t/perf-node-gather-daemonset-45xgp" Jul 28 23:03:54.317560 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:54.317426 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4d1bfed9-2800-4762-a98c-16f450c58bd3-sys\") pod \"perf-node-gather-daemonset-45xgp\" (UID: \"4d1bfed9-2800-4762-a98c-16f450c58bd3\") " pod="openshift-must-gather-tzj6t/perf-node-gather-daemonset-45xgp" Jul 28 23:03:54.317560 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:54.317452 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4d1bfed9-2800-4762-a98c-16f450c58bd3-podres\") pod \"perf-node-gather-daemonset-45xgp\" (UID: \"4d1bfed9-2800-4762-a98c-16f450c58bd3\") " pod="openshift-must-gather-tzj6t/perf-node-gather-daemonset-45xgp" Jul 28 23:03:54.317560 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:54.317472 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4d1bfed9-2800-4762-a98c-16f450c58bd3-proc\") pod \"perf-node-gather-daemonset-45xgp\" (UID: \"4d1bfed9-2800-4762-a98c-16f450c58bd3\") " pod="openshift-must-gather-tzj6t/perf-node-gather-daemonset-45xgp" Jul 28 23:03:54.317560 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:54.317477 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmxq2\" (UniqueName: \"kubernetes.io/projected/4d1bfed9-2800-4762-a98c-16f450c58bd3-kube-api-access-mmxq2\") pod \"perf-node-gather-daemonset-45xgp\" (UID: \"4d1bfed9-2800-4762-a98c-16f450c58bd3\") " pod="openshift-must-gather-tzj6t/perf-node-gather-daemonset-45xgp" Jul 28 23:03:54.317560 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:54.317536 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4d1bfed9-2800-4762-a98c-16f450c58bd3-sys\") pod \"perf-node-gather-daemonset-45xgp\" (UID: \"4d1bfed9-2800-4762-a98c-16f450c58bd3\") " pod="openshift-must-gather-tzj6t/perf-node-gather-daemonset-45xgp" Jul 28 23:03:54.317794 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:54.317573 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4d1bfed9-2800-4762-a98c-16f450c58bd3-podres\") pod \"perf-node-gather-daemonset-45xgp\" (UID: \"4d1bfed9-2800-4762-a98c-16f450c58bd3\") " pod="openshift-must-gather-tzj6t/perf-node-gather-daemonset-45xgp" Jul 28 23:03:54.317794 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:54.317583 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4d1bfed9-2800-4762-a98c-16f450c58bd3-lib-modules\") pod \"perf-node-gather-daemonset-45xgp\" (UID: \"4d1bfed9-2800-4762-a98c-16f450c58bd3\") " pod="openshift-must-gather-tzj6t/perf-node-gather-daemonset-45xgp" Jul 28 23:03:54.324952 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:54.324933 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmxq2\" (UniqueName: \"kubernetes.io/projected/4d1bfed9-2800-4762-a98c-16f450c58bd3-kube-api-access-mmxq2\") pod \"perf-node-gather-daemonset-45xgp\" (UID: \"4d1bfed9-2800-4762-a98c-16f450c58bd3\") " pod="openshift-must-gather-tzj6t/perf-node-gather-daemonset-45xgp" Jul 28 23:03:54.485185 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:54.485155 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tzj6t/perf-node-gather-daemonset-45xgp" Jul 28 23:03:54.617877 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:54.617851 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tzj6t/perf-node-gather-daemonset-45xgp"] Jul 28 23:03:54.620744 ip-10-0-142-132 kubenswrapper[2553]: W0728 23:03:54.620718 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4d1bfed9_2800_4762_a98c_16f450c58bd3.slice/crio-2ca6d740f89deb5266267c6f7f5ad4df18122a67d4e29611b4c07508fa16e205 WatchSource:0}: Error finding container 2ca6d740f89deb5266267c6f7f5ad4df18122a67d4e29611b4c07508fa16e205: Status 404 returned error can't find the container with id 2ca6d740f89deb5266267c6f7f5ad4df18122a67d4e29611b4c07508fa16e205 Jul 28 23:03:54.723249 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:54.723224 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-ncbpn_4529840b-b684-4b37-b9ab-b066f7456f1b/networking-console-plugin/0.log" Jul 28 23:03:54.767722 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:54.767617 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tzj6t/perf-node-gather-daemonset-45xgp" event={"ID":"4d1bfed9-2800-4762-a98c-16f450c58bd3","Type":"ContainerStarted","Data":"111f6e1ae153971e2b2006a0ae220457eb558d690078cee0702d06997bc07606"} Jul 28 23:03:54.767722 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:54.767674 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tzj6t/perf-node-gather-daemonset-45xgp" event={"ID":"4d1bfed9-2800-4762-a98c-16f450c58bd3","Type":"ContainerStarted","Data":"2ca6d740f89deb5266267c6f7f5ad4df18122a67d4e29611b4c07508fa16e205"} Jul 28 23:03:54.767913 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:54.767750 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-tzj6t/perf-node-gather-daemonset-45xgp" Jul 28 23:03:54.781987 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:54.781941 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tzj6t/perf-node-gather-daemonset-45xgp" podStartSLOduration=0.781926468 podStartE2EDuration="781.926468ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 23:03:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:03:54.780458957 +0000 UTC m=+3254.867717027" watchObservedRunningTime="2026-07-28 23:03:54.781926468 +0000 UTC m=+3254.869184536" Jul 28 23:03:55.316050 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:55.316016 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mldhp_e1ced996-f136-4d3e-9c85-a5b7809b4517/console-operator/1.log" Jul 28 23:03:55.320642 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:55.320619 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mldhp_e1ced996-f136-4d3e-9c85-a5b7809b4517/console-operator/2.log" Jul 28 23:03:55.856961 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:55.856917 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-hcqll_cd00fd19-681f-4de7-85ad-aea576edbac6/download-server/0.log" Jul 28 23:03:56.418688 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:56.418656 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-2dpdh_b2f1e639-88f4-4c61-96ae-3a7b694e44e1/volume-data-source-validator/0.log" Jul 28 23:03:57.473967 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:57.473939 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-l6gf5_eb1d24ec-acde-48ed-9004-aa372bc53057/dns/0.log" Jul 28 23:03:57.493450 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:57.493427 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-l6gf5_eb1d24ec-acde-48ed-9004-aa372bc53057/kube-rbac-proxy/0.log" Jul 28 23:03:57.613635 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:57.613602 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-w9bnn_0f57ff96-5ed8-4a73-9dfd-7e5d45bd163c/dns-node-resolver/0.log" Jul 28 23:03:58.218439 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:58.218413 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bs64q_54dc150c-a1f7-4df4-9adf-03077bf59073/node-ca/0.log" Jul 28 23:03:59.338856 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:59.338826 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-vdkjj_c283e4ed-82e2-474a-b100-c34afa239b17/discovery/0.log" Jul 28 23:03:59.428173 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:59.428141 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-bfrsf_e776e265-c83c-42be-be8c-07647513e2ba/istio-proxy/0.log" Jul 28 23:03:59.457222 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:59.457196 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-bfrsf_e776e265-c83c-42be-be8c-07647513e2ba/istio-proxy/1.log" Jul 28 23:03:59.568719 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:03:59.568684 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6747b7576c-fs2bd_7e266916-99eb-4807-9b85-fabdeb9e1c28/router/0.log" Jul 28 23:04:00.377149 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:04:00.377121 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-q8wn6_8cea718f-40c0-48b7-943c-7c4178bd8d0d/serve-healthcheck-canary/0.log" Jul 28 23:04:00.782423 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:04:00.782390 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-tzj6t/perf-node-gather-daemonset-45xgp" Jul 28 23:04:00.895880 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:04:00.895852 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-bpt84_414a9396-e46b-433a-9976-95c5c0b1d897/insights-operator/0.log" Jul 28 23:04:00.896931 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:04:00.896914 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-bpt84_414a9396-e46b-433a-9976-95c5c0b1d897/insights-operator/1.log" Jul 28 23:04:01.055435 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:04:01.055356 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wx69k_4c004d0a-52da-428b-b2f3-aefe98d15909/kube-rbac-proxy/0.log" Jul 28 23:04:01.075971 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:04:01.075943 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wx69k_4c004d0a-52da-428b-b2f3-aefe98d15909/exporter/0.log" Jul 28 23:04:01.096840 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:04:01.096812 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wx69k_4c004d0a-52da-428b-b2f3-aefe98d15909/extractor/0.log" Jul 28 23:04:04.028086 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:04:04.028053 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-d6457fd9-dghbg_4184e9a3-2f74-4275-8c85-5764766290cd/manager/0.log" Jul 28 23:04:04.281753 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:04:04.281696 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b7ccc5769-88sx5_331d730d-2853-4509-9e90-4727ad3ed40d/manager/0.log" Jul 28 23:04:04.304188 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:04:04.304166 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b7ccc5769-88sx5_331d730d-2853-4509-9e90-4727ad3ed40d/copy-manifests/0.log" Jul 28 23:04:04.873954 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:04:04.873921 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-vdkjj_c283e4ed-82e2-474a-b100-c34afa239b17/discovery/0.log" Jul 28 23:04:04.923595 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:04:04.923566 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-bfrsf_e776e265-c83c-42be-be8c-07647513e2ba/istio-proxy/0.log" Jul 28 23:04:04.952380 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:04:04.952359 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-bfrsf_e776e265-c83c-42be-be8c-07647513e2ba/istio-proxy/1.log" Jul 28 23:04:05.024788 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:04:05.024759 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6747b7576c-fs2bd_7e266916-99eb-4807-9b85-fabdeb9e1c28/router/0.log" Jul 28 23:04:05.794584 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:04:05.794555 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-8586db4f6f-rqqn9_fae924a4-7ded-4b72-a773-beb3d4e18e84/manager/0.log" Jul 28 23:04:06.613678 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:04:06.613628 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-vdkjj_c283e4ed-82e2-474a-b100-c34afa239b17/discovery/0.log" Jul 28 23:04:06.657579 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:04:06.657550 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-bfrsf_e776e265-c83c-42be-be8c-07647513e2ba/istio-proxy/0.log" Jul 28 23:04:06.684379 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:04:06.684353 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-bfrsf_e776e265-c83c-42be-be8c-07647513e2ba/istio-proxy/1.log" Jul 28 23:04:06.764020 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:04:06.763993 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6747b7576c-fs2bd_7e266916-99eb-4807-9b85-fabdeb9e1c28/router/0.log" Jul 28 23:04:10.558143 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:04:10.558114 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-x7ks4_fc8b5f1d-2ee7-4775-8ee8-cc1175781678/migrator/0.log" Jul 28 23:04:10.578918 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:04:10.578895 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-x7ks4_fc8b5f1d-2ee7-4775-8ee8-cc1175781678/graceful-termination/0.log" Jul 28 23:04:10.939839 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:04:10.939762 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-nzswh_c1aff3e5-115b-4502-899a-b2137168973b/kube-storage-version-migrator-operator/1.log" Jul 28 23:04:10.940756 ip-10-0-142-132 kubenswrapper[2553]: I0728 23:04:10.940695 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-nzswh_c1aff3e5-115b-4502-899a-b2137168973b/kube-storage-version-migrator-operator/0.log"