Jul 13 21:57:46.787748 ip-10-0-133-234 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 13 21:57:46.787757 ip-10-0-133-234 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 13 21:57:46.787764 ip-10-0-133-234 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 13 21:57:46.788015 ip-10-0-133-234 systemd[1]: Failed to start Kubernetes Kubelet. Jul 13 21:57:56.891694 ip-10-0-133-234 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 13 21:57:56.891714 ip-10-0-133-234 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 3eeb3d2149194e16a867f84a18c0cf68 -- Jul 13 21:59:52.655045 ip-10-0-133-234 systemd[1]: Starting Kubernetes Kubelet... Jul 13 21:59:53.091139 ip-10-0-133-234 kubenswrapper[2555]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 13 21:59:53.091139 ip-10-0-133-234 kubenswrapper[2555]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 13 21:59:53.091139 ip-10-0-133-234 kubenswrapper[2555]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 13 21:59:53.091139 ip-10-0-133-234 kubenswrapper[2555]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 13 21:59:53.091139 ip-10-0-133-234 kubenswrapper[2555]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 13 21:59:53.092308 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.092187 2555 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 13 21:59:53.098591 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098573 2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 21:59:53.098591 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098588 2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 21:59:53.098591 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098592 2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 21:59:53.098591 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098595 2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 21:59:53.098591 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098598 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 21:59:53.098786 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098601 2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 21:59:53.098786 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098604 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 21:59:53.098786 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098606 2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 21:59:53.098786 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098609 2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 21:59:53.098786 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098612 2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 21:59:53.098786 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098614 2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 21:59:53.098786 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098617 2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 21:59:53.098786 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098619 2555 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 21:59:53.098786 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098622 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 21:59:53.098786 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098625 2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 21:59:53.098786 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098627 2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 21:59:53.098786 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098630 2555 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 21:59:53.098786 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098632 2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 21:59:53.098786 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098635 2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 21:59:53.098786 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098638 2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 21:59:53.098786 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098641 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 21:59:53.098786 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098643 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 21:59:53.098786 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098646 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 21:59:53.098786 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098648 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 21:59:53.098786 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098651 2555 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 21:59:53.098786 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098653 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 21:59:53.099297 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098656 2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 21:59:53.099297 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098658 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 21:59:53.099297 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098661 2555 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 21:59:53.099297 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098663 2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 21:59:53.099297 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098665 2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 21:59:53.099297 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098668 2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 21:59:53.099297 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098670 2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 21:59:53.099297 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098673 2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 21:59:53.099297 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098675 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 21:59:53.099297 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098678 2555 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 21:59:53.099297 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098680 2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 21:59:53.099297 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098682 2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 21:59:53.099297 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098687 2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 21:59:53.099297 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098690 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 21:59:53.099297 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098692 2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 21:59:53.099297 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098694 2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 21:59:53.099297 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098697 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 21:59:53.099297 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098700 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 21:59:53.099297 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098702 2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 21:59:53.099297 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098706 2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 21:59:53.099812 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098708 2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 21:59:53.099812 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098710 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 21:59:53.099812 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098713 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 21:59:53.099812 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098715 2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 21:59:53.099812 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098718 2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 21:59:53.099812 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098720 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 21:59:53.099812 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098723 2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 21:59:53.099812 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098727 2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 21:59:53.099812 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098730 2555 feature_gate.go:328] unrecognized feature gate: Example Jul 13 21:59:53.099812 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098733 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 21:59:53.099812 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098735 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 21:59:53.099812 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098738 2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 21:59:53.099812 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098740 2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 21:59:53.099812 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098743 2555 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 21:59:53.099812 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098745 2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 21:59:53.099812 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098748 2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 21:59:53.099812 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098750 2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 21:59:53.099812 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098752 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 21:59:53.099812 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098755 2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 21:59:53.100276 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098757 2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 21:59:53.100276 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098760 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 21:59:53.100276 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098762 2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 21:59:53.100276 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098765 2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 21:59:53.100276 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098769 2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 21:59:53.100276 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098787 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 21:59:53.100276 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098791 2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 21:59:53.100276 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098794 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 21:59:53.100276 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098797 2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 21:59:53.100276 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098800 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 21:59:53.100276 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098805 2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 21:59:53.100276 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098808 2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 21:59:53.100276 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098810 2555 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 21:59:53.100276 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098814 2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 21:59:53.100276 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098816 2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 21:59:53.100276 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098820 2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 21:59:53.100276 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098822 2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 21:59:53.100276 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098825 2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 21:59:53.100276 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098828 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 21:59:53.100749 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098830 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 21:59:53.100749 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098833 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 21:59:53.100749 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098835 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 21:59:53.100749 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098837 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 21:59:53.100749 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098840 2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 21:59:53.100749 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098842 2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 21:59:53.100749 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098845 2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 21:59:53.100749 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098847 2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 21:59:53.100749 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098850 2555 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 21:59:53.100749 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098852 2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 21:59:53.100749 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098856 2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 21:59:53.100749 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098860 2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 21:59:53.100749 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098863 2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 21:59:53.100749 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098866 2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 21:59:53.100749 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.098869 2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 21:59:53.100749 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099566 2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 21:59:53.100749 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099572 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 21:59:53.100749 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099575 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 21:59:53.100749 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099579 2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 21:59:53.101326 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099581 2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 21:59:53.101326 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099586 2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 21:59:53.101326 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099590 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 21:59:53.101326 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099593 2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 21:59:53.101326 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099596 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 21:59:53.101326 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099598 2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 21:59:53.101326 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099601 2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 21:59:53.101326 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099604 2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 21:59:53.101326 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099606 2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 21:59:53.101326 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099609 2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 21:59:53.101326 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099611 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 21:59:53.101326 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099613 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 21:59:53.101326 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099616 2555 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 21:59:53.101326 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099619 2555 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 21:59:53.101326 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099621 2555 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 21:59:53.101326 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099623 2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 21:59:53.101326 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099626 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 21:59:53.101326 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099628 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 21:59:53.101326 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099631 2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 21:59:53.101326 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099633 2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 21:59:53.101879 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099636 2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 21:59:53.101879 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099638 2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 21:59:53.101879 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099640 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 21:59:53.101879 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099643 2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 21:59:53.101879 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099646 2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 21:59:53.101879 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099648 2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 21:59:53.101879 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099651 2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 21:59:53.101879 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099653 2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 21:59:53.101879 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099656 2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 21:59:53.101879 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099658 2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 21:59:53.101879 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099661 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 21:59:53.101879 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099664 2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 21:59:53.101879 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099666 2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 21:59:53.101879 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099669 2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 21:59:53.101879 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099672 2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 21:59:53.101879 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099674 2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 21:59:53.101879 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099676 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 21:59:53.101879 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099678 2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 21:59:53.101879 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099681 2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 21:59:53.101879 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099683 2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 21:59:53.102377 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099685 2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 21:59:53.102377 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099688 2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 21:59:53.102377 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099690 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 21:59:53.102377 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099693 2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 21:59:53.102377 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099696 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 21:59:53.102377 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099698 2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 21:59:53.102377 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099701 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 21:59:53.102377 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099703 2555 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 21:59:53.102377 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099705 2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 21:59:53.102377 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099708 2555 feature_gate.go:328] unrecognized feature gate: Example Jul 13 21:59:53.102377 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099710 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 21:59:53.102377 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099714 2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 21:59:53.102377 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099718 2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 21:59:53.102377 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099721 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 21:59:53.102377 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099723 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 21:59:53.102377 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099726 2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 21:59:53.102377 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099729 2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 21:59:53.102377 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099732 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 21:59:53.102377 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099734 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 21:59:53.102863 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099737 2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 21:59:53.102863 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099739 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 21:59:53.102863 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099742 2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 21:59:53.102863 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099744 2555 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 21:59:53.102863 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099747 2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 21:59:53.102863 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099750 2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 21:59:53.102863 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099752 2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 21:59:53.102863 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099754 2555 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 21:59:53.102863 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099757 2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 21:59:53.102863 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099759 2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 21:59:53.102863 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099762 2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 21:59:53.102863 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099764 2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 21:59:53.102863 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099767 2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 21:59:53.102863 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099769 2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 21:59:53.102863 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099787 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 21:59:53.102863 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099789 2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 21:59:53.102863 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099793 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 21:59:53.102863 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099796 2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 21:59:53.102863 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099798 2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 21:59:53.103321 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099802 2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 21:59:53.103321 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099805 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 21:59:53.103321 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099809 2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 21:59:53.103321 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099812 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 21:59:53.103321 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099814 2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 21:59:53.103321 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099817 2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 21:59:53.103321 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099820 2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 21:59:53.103321 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099822 2555 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 21:59:53.103321 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099825 2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 21:59:53.103321 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099828 2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 21:59:53.103321 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099831 2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 21:59:53.103321 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099833 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 21:59:53.103321 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099836 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 21:59:53.103321 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099839 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 21:59:53.103321 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099841 2555 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 21:59:53.103321 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099843 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 21:59:53.103321 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.099846 2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 21:59:53.103321 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100194 2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 21:59:53.103321 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100199 2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 21:59:53.103812 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100201 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 21:59:53.103812 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100204 2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 21:59:53.103812 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100207 2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 21:59:53.103812 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100209 2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 21:59:53.103812 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100212 2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 21:59:53.103812 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100214 2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 21:59:53.103812 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100217 2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 21:59:53.103812 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100220 2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 21:59:53.103812 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100222 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 21:59:53.103812 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100225 2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 21:59:53.103812 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100227 2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 21:59:53.103812 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100229 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 21:59:53.103812 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100232 2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 21:59:53.103812 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100234 2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 21:59:53.103812 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100236 2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 21:59:53.103812 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100239 2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 21:59:53.103812 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100242 2555 feature_gate.go:328] unrecognized feature gate: Example Jul 13 21:59:53.103812 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100244 2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 21:59:53.103812 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100246 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 21:59:53.103812 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100249 2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 21:59:53.104308 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100251 2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 21:59:53.104308 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100254 2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 21:59:53.104308 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100256 2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 21:59:53.104308 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100259 2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 21:59:53.104308 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100262 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 21:59:53.104308 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100264 2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 21:59:53.104308 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100267 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 21:59:53.104308 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100269 2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 21:59:53.104308 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100272 2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 21:59:53.104308 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100276 2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 21:59:53.104308 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100279 2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 21:59:53.104308 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100282 2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 21:59:53.104308 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100284 2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 21:59:53.104308 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100286 2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 21:59:53.104308 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100289 2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 21:59:53.104308 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100292 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 21:59:53.104308 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100295 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 21:59:53.104308 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100299 2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 21:59:53.104308 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100302 2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 21:59:53.104814 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100305 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 21:59:53.104814 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100308 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 21:59:53.104814 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100310 2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 21:59:53.104814 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100313 2555 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 21:59:53.104814 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100316 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 21:59:53.104814 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100319 2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 21:59:53.104814 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100322 2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 21:59:53.104814 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100325 2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 21:59:53.104814 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100328 2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 21:59:53.104814 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100330 2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 21:59:53.104814 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100332 2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 21:59:53.104814 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100335 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 21:59:53.104814 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100337 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 21:59:53.104814 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100340 2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 21:59:53.104814 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100342 2555 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 21:59:53.104814 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100345 2555 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 21:59:53.104814 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100350 2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 21:59:53.104814 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100352 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 21:59:53.104814 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100355 2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 21:59:53.104814 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100357 2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 21:59:53.105308 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100359 2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 21:59:53.105308 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100362 2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 21:59:53.105308 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100365 2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 21:59:53.105308 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100368 2555 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 21:59:53.105308 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100371 2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 21:59:53.105308 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100373 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 21:59:53.105308 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100376 2555 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 21:59:53.105308 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100378 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 21:59:53.105308 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100381 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 21:59:53.105308 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100384 2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 21:59:53.105308 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100386 2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 21:59:53.105308 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100389 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 21:59:53.105308 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100392 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 21:59:53.105308 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100394 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 21:59:53.105308 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100396 2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 21:59:53.105308 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100399 2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 21:59:53.105308 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100401 2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 21:59:53.105308 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100404 2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 21:59:53.105308 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100407 2555 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 21:59:53.105308 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100409 2555 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 21:59:53.105823 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100411 2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 21:59:53.105823 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100415 2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 21:59:53.105823 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100418 2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 21:59:53.105823 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100421 2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 21:59:53.105823 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100424 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 21:59:53.105823 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100427 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 21:59:53.105823 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100429 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 21:59:53.105823 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100432 2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 21:59:53.105823 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100436 2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 21:59:53.105823 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100438 2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 21:59:53.105823 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100441 2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 21:59:53.105823 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100443 2555 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 21:59:53.105823 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100446 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 21:59:53.105823 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100448 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 21:59:53.105823 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100452 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 21:59:53.105823 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100454 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 21:59:53.105823 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100457 2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 21:59:53.105823 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.100459 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 21:59:53.105823 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100527 2555 flags.go:64] FLAG: --address="0.0.0.0" Jul 13 21:59:53.105823 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100536 2555 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 13 21:59:53.106304 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100544 2555 flags.go:64] FLAG: --anonymous-auth="true" Jul 13 21:59:53.106304 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100551 2555 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 13 21:59:53.106304 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100557 2555 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 13 21:59:53.106304 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100562 2555 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 13 21:59:53.106304 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100568 2555 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 13 21:59:53.106304 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100574 2555 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 13 21:59:53.106304 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100579 2555 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 13 21:59:53.106304 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100583 2555 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 13 21:59:53.106304 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100586 2555 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 13 21:59:53.106304 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100589 2555 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 13 21:59:53.106304 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100592 2555 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 13 21:59:53.106304 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100595 2555 flags.go:64] FLAG: --cgroup-root="" Jul 13 21:59:53.106304 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100598 2555 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 13 21:59:53.106304 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100601 2555 flags.go:64] FLAG: --client-ca-file="" Jul 13 21:59:53.106304 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100603 2555 flags.go:64] FLAG: --cloud-provider="external" Jul 13 21:59:53.106304 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100606 2555 flags.go:64] FLAG: --cluster-dns="[]" Jul 13 21:59:53.106304 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100611 2555 flags.go:64] FLAG: --cluster-domain="" Jul 13 21:59:53.106304 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100614 2555 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 13 21:59:53.106304 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100617 2555 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 13 21:59:53.106304 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100620 2555 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 13 21:59:53.106304 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100624 2555 flags.go:64] FLAG: --container-log-max-files="5" Jul 13 21:59:53.106304 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100628 2555 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 13 21:59:53.106304 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100631 2555 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 13 21:59:53.106913 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100634 2555 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 13 21:59:53.106913 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100637 2555 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 13 21:59:53.106913 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100640 2555 flags.go:64] FLAG: --contention-profiling="false" Jul 13 21:59:53.106913 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100644 2555 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 13 21:59:53.106913 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100647 2555 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 13 21:59:53.106913 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100650 2555 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 13 21:59:53.106913 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100653 2555 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 13 21:59:53.106913 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100657 2555 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 13 21:59:53.106913 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100660 2555 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 13 21:59:53.106913 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100669 2555 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 13 21:59:53.106913 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100673 2555 flags.go:64] FLAG: --enable-load-reader="false" Jul 13 21:59:53.106913 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100677 2555 flags.go:64] FLAG: --enable-server="true" Jul 13 21:59:53.106913 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100680 2555 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 13 21:59:53.106913 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100685 2555 flags.go:64] FLAG: --event-burst="100" Jul 13 21:59:53.106913 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100688 2555 flags.go:64] FLAG: --event-qps="50" Jul 13 21:59:53.106913 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100690 2555 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 13 21:59:53.106913 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100693 2555 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 13 21:59:53.106913 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100696 2555 flags.go:64] FLAG: --eviction-hard="" Jul 13 21:59:53.106913 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100700 2555 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 13 21:59:53.106913 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100702 2555 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 13 21:59:53.106913 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100705 2555 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 13 21:59:53.106913 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100708 2555 flags.go:64] FLAG: --eviction-soft="" Jul 13 21:59:53.106913 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100711 2555 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 13 21:59:53.106913 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100714 2555 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 13 21:59:53.106913 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100716 2555 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 13 21:59:53.107534 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100719 2555 flags.go:64] FLAG: --experimental-mounter-path="" Jul 13 21:59:53.107534 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100722 2555 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 13 21:59:53.107534 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100725 2555 flags.go:64] FLAG: --fail-swap-on="true" Jul 13 21:59:53.107534 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100727 2555 flags.go:64] FLAG: --feature-gates="" Jul 13 21:59:53.107534 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100733 2555 flags.go:64] FLAG: --file-check-frequency="20s" Jul 13 21:59:53.107534 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100736 2555 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 13 21:59:53.107534 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100738 2555 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 13 21:59:53.107534 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100741 2555 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 13 21:59:53.107534 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100744 2555 flags.go:64] FLAG: --healthz-port="10248" Jul 13 21:59:53.107534 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100748 2555 flags.go:64] FLAG: --help="false" Jul 13 21:59:53.107534 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100752 2555 flags.go:64] FLAG: --hostname-override="ip-10-0-133-234.ec2.internal" Jul 13 21:59:53.107534 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100755 2555 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 13 21:59:53.107534 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100759 2555 flags.go:64] FLAG: --http-check-frequency="20s" Jul 13 21:59:53.107534 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100762 2555 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 13 21:59:53.107534 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100765 2555 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 13 21:59:53.107534 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100768 2555 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 13 21:59:53.107534 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100771 2555 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 13 21:59:53.107534 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100790 2555 flags.go:64] FLAG: --image-service-endpoint="" Jul 13 21:59:53.107534 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100793 2555 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 13 21:59:53.107534 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100796 2555 flags.go:64] FLAG: --kube-api-burst="100" Jul 13 21:59:53.107534 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100799 2555 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 13 21:59:53.107534 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100802 2555 flags.go:64] FLAG: --kube-api-qps="50" Jul 13 21:59:53.107534 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100805 2555 flags.go:64] FLAG: --kube-reserved="" Jul 13 21:59:53.107534 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100807 2555 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 13 21:59:53.108191 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100810 2555 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 13 21:59:53.108191 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100813 2555 flags.go:64] FLAG: --kubelet-cgroups="" Jul 13 21:59:53.108191 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100816 2555 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 13 21:59:53.108191 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100819 2555 flags.go:64] FLAG: --lock-file="" Jul 13 21:59:53.108191 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100821 2555 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 13 21:59:53.108191 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100824 2555 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 13 21:59:53.108191 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100827 2555 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 13 21:59:53.108191 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100832 2555 flags.go:64] FLAG: --log-json-split-stream="false" Jul 13 21:59:53.108191 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100834 2555 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 13 21:59:53.108191 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100837 2555 flags.go:64] FLAG: --log-text-split-stream="false" Jul 13 21:59:53.108191 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100840 2555 flags.go:64] FLAG: --logging-format="text" Jul 13 21:59:53.108191 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100843 2555 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 13 21:59:53.108191 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100847 2555 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 13 21:59:53.108191 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100850 2555 flags.go:64] FLAG: --manifest-url="" Jul 13 21:59:53.108191 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100852 2555 flags.go:64] FLAG: --manifest-url-header="" Jul 13 21:59:53.108191 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100856 2555 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 13 21:59:53.108191 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100859 2555 flags.go:64] FLAG: --max-open-files="1000000" Jul 13 21:59:53.108191 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100863 2555 flags.go:64] FLAG: --max-pods="110" Jul 13 21:59:53.108191 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100867 2555 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 13 21:59:53.108191 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100870 2555 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 13 21:59:53.108191 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100872 2555 flags.go:64] FLAG: --memory-manager-policy="None" Jul 13 21:59:53.108191 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100875 2555 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 13 21:59:53.108191 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100878 2555 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 13 21:59:53.108191 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100881 2555 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 13 21:59:53.108191 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100884 2555 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 13 21:59:53.108908 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100890 2555 flags.go:64] FLAG: --node-status-max-images="50" Jul 13 21:59:53.108908 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100893 2555 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 13 21:59:53.108908 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100896 2555 flags.go:64] FLAG: --oom-score-adj="-999" Jul 13 21:59:53.108908 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100899 2555 flags.go:64] FLAG: --pod-cidr="" Jul 13 21:59:53.108908 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100903 2555 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 13 21:59:53.108908 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100907 2555 flags.go:64] FLAG: --pod-manifest-path="" Jul 13 21:59:53.108908 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100910 2555 flags.go:64] FLAG: --pod-max-pids="-1" Jul 13 21:59:53.108908 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100913 2555 flags.go:64] FLAG: --pods-per-core="0" Jul 13 21:59:53.108908 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100916 2555 flags.go:64] FLAG: --port="10250" Jul 13 21:59:53.108908 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100919 2555 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 13 21:59:53.108908 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100922 2555 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0443025d6c2fc7799" Jul 13 21:59:53.108908 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100925 2555 flags.go:64] FLAG: --qos-reserved="" Jul 13 21:59:53.108908 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100928 2555 flags.go:64] FLAG: --read-only-port="10255" Jul 13 21:59:53.108908 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100931 2555 flags.go:64] FLAG: --register-node="true" Jul 13 21:59:53.108908 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100934 2555 flags.go:64] FLAG: --register-with-taints="" Jul 13 21:59:53.108908 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100938 2555 flags.go:64] FLAG: --registry-burst="10" Jul 13 21:59:53.108908 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100940 2555 flags.go:64] FLAG: --registry-qps="5" Jul 13 21:59:53.108908 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100943 2555 flags.go:64] FLAG: --reserved-cpus="" Jul 13 21:59:53.108908 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100945 2555 flags.go:64] FLAG: --reserved-memory="" Jul 13 21:59:53.108908 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100949 2555 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 13 21:59:53.108908 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100953 2555 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 13 21:59:53.108908 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100956 2555 flags.go:64] FLAG: --rotate-certificates="false" Jul 13 21:59:53.108908 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100959 2555 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 13 21:59:53.108908 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100962 2555 flags.go:64] FLAG: --runonce="false" Jul 13 21:59:53.108908 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100964 2555 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 13 21:59:53.109508 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100968 2555 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 13 21:59:53.109508 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100971 2555 flags.go:64] FLAG: --seccomp-default="false" Jul 13 21:59:53.109508 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100973 2555 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 13 21:59:53.109508 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100976 2555 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 13 21:59:53.109508 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100979 2555 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 13 21:59:53.109508 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100983 2555 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 13 21:59:53.109508 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100986 2555 flags.go:64] FLAG: --storage-driver-password="root" Jul 13 21:59:53.109508 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100989 2555 flags.go:64] FLAG: --storage-driver-secure="false" Jul 13 21:59:53.109508 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100992 2555 flags.go:64] FLAG: --storage-driver-table="stats" Jul 13 21:59:53.109508 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100994 2555 flags.go:64] FLAG: --storage-driver-user="root" Jul 13 21:59:53.109508 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.100997 2555 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 13 21:59:53.109508 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.101000 2555 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 13 21:59:53.109508 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.101004 2555 flags.go:64] FLAG: --system-cgroups="" Jul 13 21:59:53.109508 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.101007 2555 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 13 21:59:53.109508 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.101012 2555 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 13 21:59:53.109508 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.101015 2555 flags.go:64] FLAG: --tls-cert-file="" Jul 13 21:59:53.109508 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.101018 2555 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 13 21:59:53.109508 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.101022 2555 flags.go:64] FLAG: --tls-min-version="" Jul 13 21:59:53.109508 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.101025 2555 flags.go:64] FLAG: --tls-private-key-file="" Jul 13 21:59:53.109508 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.101028 2555 flags.go:64] FLAG: --topology-manager-policy="none" Jul 13 21:59:53.109508 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.101030 2555 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 13 21:59:53.109508 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.101033 2555 flags.go:64] FLAG: --topology-manager-scope="container" Jul 13 21:59:53.109508 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.101036 2555 flags.go:64] FLAG: --v="2" Jul 13 21:59:53.109508 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.101039 2555 flags.go:64] FLAG: --version="false" Jul 13 21:59:53.109508 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.101043 2555 flags.go:64] FLAG: --vmodule="" Jul 13 21:59:53.110135 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.101046 2555 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 13 21:59:53.110135 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.101049 2555 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 13 21:59:53.110135 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101166 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 21:59:53.110135 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101171 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 21:59:53.110135 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101174 2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 21:59:53.110135 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101177 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 21:59:53.110135 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101181 2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 21:59:53.110135 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101184 2555 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 21:59:53.110135 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101187 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 21:59:53.110135 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101189 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 21:59:53.110135 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101192 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 21:59:53.110135 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101195 2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 21:59:53.110135 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101197 2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 21:59:53.110135 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101200 2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 21:59:53.110135 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101203 2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 21:59:53.110135 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101205 2555 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 21:59:53.110135 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101208 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 21:59:53.110135 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101210 2555 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 21:59:53.110135 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101213 2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 21:59:53.110135 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101216 2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 21:59:53.110660 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101224 2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 21:59:53.110660 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101227 2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 21:59:53.110660 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101230 2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 21:59:53.110660 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101232 2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 21:59:53.110660 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101234 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 21:59:53.110660 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101237 2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 21:59:53.110660 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101239 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 21:59:53.110660 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101242 2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 21:59:53.110660 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101245 2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 21:59:53.110660 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101247 2555 feature_gate.go:328] unrecognized feature gate: Example Jul 13 21:59:53.110660 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101249 2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 21:59:53.110660 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101252 2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 21:59:53.110660 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101254 2555 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 21:59:53.110660 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101257 2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 21:59:53.110660 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101260 2555 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 21:59:53.110660 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101263 2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 21:59:53.110660 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101266 2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 21:59:53.110660 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101269 2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 21:59:53.110660 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101273 2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 21:59:53.110660 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101275 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 21:59:53.111159 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101278 2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 21:59:53.111159 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101280 2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 21:59:53.111159 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101283 2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 21:59:53.111159 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101285 2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 21:59:53.111159 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101288 2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 21:59:53.111159 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101290 2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 21:59:53.111159 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101292 2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 21:59:53.111159 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101295 2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 21:59:53.111159 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101297 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 21:59:53.111159 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101299 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 21:59:53.111159 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101302 2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 21:59:53.111159 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101304 2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 21:59:53.111159 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101307 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 21:59:53.111159 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101316 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 21:59:53.111159 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101318 2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 21:59:53.111159 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101321 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 21:59:53.111159 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101323 2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 21:59:53.111159 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101326 2555 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 21:59:53.111159 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101328 2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 21:59:53.111627 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101331 2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 21:59:53.111627 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101333 2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 21:59:53.111627 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101337 2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 21:59:53.111627 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101341 2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 21:59:53.111627 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101344 2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 21:59:53.111627 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101347 2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 21:59:53.111627 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101350 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 21:59:53.111627 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101354 2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 21:59:53.111627 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101357 2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 21:59:53.111627 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101359 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 21:59:53.111627 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101362 2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 21:59:53.111627 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101365 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 21:59:53.111627 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101368 2555 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 21:59:53.111627 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101371 2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 21:59:53.111627 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101373 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 21:59:53.111627 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101375 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 21:59:53.111627 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101378 2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 21:59:53.111627 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101380 2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 21:59:53.111627 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101382 2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 21:59:53.112113 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101385 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 21:59:53.112113 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101387 2555 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 21:59:53.112113 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101389 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 21:59:53.112113 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101392 2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 21:59:53.112113 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101394 2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 21:59:53.112113 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101397 2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 21:59:53.112113 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101399 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 21:59:53.112113 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101402 2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 21:59:53.112113 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101405 2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 21:59:53.112113 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101407 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 21:59:53.112113 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101410 2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 21:59:53.112113 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101412 2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 21:59:53.112113 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101414 2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 21:59:53.112113 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101416 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 21:59:53.112113 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101419 2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 21:59:53.112113 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101421 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 21:59:53.112113 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101424 2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 21:59:53.112113 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101426 2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 21:59:53.112113 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101428 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 21:59:53.112113 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101431 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 21:59:53.112596 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101435 2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 21:59:53.112596 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101438 2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 21:59:53.112596 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.101441 2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 21:59:53.112596 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.102349 2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 13 21:59:53.112596 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.109464 2555 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 13 21:59:53.112596 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.109480 2555 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 13 21:59:53.112596 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.110423 2555 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 13 21:59:53.112596 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.110431 2555 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 13 21:59:53.112596 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110491 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 21:59:53.112596 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110495 2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 21:59:53.112596 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110499 2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 21:59:53.112596 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110502 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 21:59:53.112596 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110505 2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 21:59:53.112596 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110507 2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 21:59:53.113034 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110510 2555 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 21:59:53.113034 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110512 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 21:59:53.113034 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110516 2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 21:59:53.113034 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110520 2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 21:59:53.113034 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110523 2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 21:59:53.113034 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110526 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 21:59:53.113034 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110529 2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 21:59:53.113034 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110531 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 21:59:53.113034 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110534 2555 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 21:59:53.113034 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110537 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 21:59:53.113034 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110539 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 21:59:53.113034 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110542 2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 21:59:53.113034 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110544 2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 21:59:53.113034 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110546 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 21:59:53.113034 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110549 2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 21:59:53.113034 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110551 2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 21:59:53.113034 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110553 2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 21:59:53.113034 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110556 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 21:59:53.113034 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110558 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 21:59:53.113034 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110561 2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 21:59:53.113566 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110563 2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 21:59:53.113566 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110566 2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 21:59:53.113566 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110568 2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 21:59:53.113566 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110577 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 21:59:53.113566 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110580 2555 feature_gate.go:328] unrecognized feature gate: Example Jul 13 21:59:53.113566 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110582 2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 21:59:53.113566 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110585 2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 21:59:53.113566 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110587 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 21:59:53.113566 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110590 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 21:59:53.113566 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110592 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 21:59:53.113566 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110594 2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 21:59:53.113566 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110597 2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 21:59:53.113566 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110599 2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 21:59:53.113566 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110602 2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 21:59:53.113566 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110604 2555 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 21:59:53.113566 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110607 2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 21:59:53.113566 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110609 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 21:59:53.113566 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110611 2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 21:59:53.113566 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110614 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 21:59:53.113566 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110616 2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 21:59:53.114093 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110618 2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 21:59:53.114093 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110621 2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 21:59:53.114093 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110623 2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 21:59:53.114093 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110625 2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 21:59:53.114093 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110628 2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 21:59:53.114093 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110630 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 21:59:53.114093 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110633 2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 21:59:53.114093 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110635 2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 21:59:53.114093 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110639 2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 21:59:53.114093 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110643 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 21:59:53.114093 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110646 2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 21:59:53.114093 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110649 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 21:59:53.114093 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110651 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 21:59:53.114093 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110654 2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 21:59:53.114093 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110657 2555 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 21:59:53.114093 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110660 2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 21:59:53.114093 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110668 2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 21:59:53.114093 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110671 2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 21:59:53.114093 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110673 2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 21:59:53.114559 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110676 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 21:59:53.114559 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110680 2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 21:59:53.114559 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110683 2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 21:59:53.114559 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110685 2555 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 21:59:53.114559 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110688 2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 21:59:53.114559 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110691 2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 21:59:53.114559 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110693 2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 21:59:53.114559 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110696 2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 21:59:53.114559 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110699 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 21:59:53.114559 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110701 2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 21:59:53.114559 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110703 2555 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 21:59:53.114559 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110706 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 21:59:53.114559 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110708 2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 21:59:53.114559 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110710 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 21:59:53.114559 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110713 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 21:59:53.114559 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110715 2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 21:59:53.114559 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110718 2555 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 21:59:53.114559 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110720 2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 21:59:53.114559 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110723 2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 21:59:53.114559 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110725 2555 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 21:59:53.115069 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110727 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 21:59:53.115069 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110730 2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 21:59:53.115069 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110732 2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 21:59:53.115069 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110734 2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 21:59:53.115069 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110737 2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 21:59:53.115069 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110739 2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 21:59:53.115069 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110742 2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 21:59:53.115069 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110744 2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 21:59:53.115069 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110747 2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 21:59:53.115069 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110749 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 21:59:53.115069 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110758 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 21:59:53.115069 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110760 2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 21:59:53.115069 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110763 2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 21:59:53.115069 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110765 2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 21:59:53.115069 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.110770 2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 13 21:59:53.115464 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110923 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 21:59:53.115464 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110929 2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 21:59:53.115464 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110932 2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 21:59:53.115464 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110936 2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 21:59:53.115464 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110939 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 21:59:53.115464 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110942 2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 21:59:53.115464 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110945 2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 21:59:53.115464 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110947 2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 21:59:53.115464 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110950 2555 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 21:59:53.115464 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110952 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 21:59:53.115464 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110955 2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 21:59:53.115464 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110957 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 21:59:53.115464 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110960 2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 21:59:53.115464 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110962 2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 21:59:53.115464 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110964 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 21:59:53.115464 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110967 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 21:59:53.115464 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110969 2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 21:59:53.115464 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110971 2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 21:59:53.115464 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110974 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 21:59:53.115948 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110978 2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 21:59:53.115948 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110981 2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 21:59:53.115948 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110984 2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 21:59:53.115948 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110986 2555 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 21:59:53.115948 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110989 2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 21:59:53.115948 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110992 2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 21:59:53.115948 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110995 2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 21:59:53.115948 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.110997 2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 21:59:53.115948 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111000 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 21:59:53.115948 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111007 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 21:59:53.115948 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111010 2555 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 21:59:53.115948 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111012 2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 21:59:53.115948 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111015 2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 21:59:53.115948 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111017 2555 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 21:59:53.115948 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111019 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 21:59:53.115948 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111022 2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 21:59:53.115948 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111024 2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 21:59:53.115948 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111027 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 21:59:53.115948 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111029 2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 21:59:53.115948 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111031 2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 21:59:53.116431 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111034 2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 21:59:53.116431 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111036 2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 21:59:53.116431 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111038 2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 21:59:53.116431 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111041 2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 21:59:53.116431 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111043 2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 21:59:53.116431 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111047 2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 21:59:53.116431 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111050 2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 21:59:53.116431 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111053 2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 21:59:53.116431 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111056 2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 21:59:53.116431 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111058 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 21:59:53.116431 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111061 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 21:59:53.116431 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111063 2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 21:59:53.116431 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111066 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 21:59:53.116431 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111069 2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 21:59:53.116431 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111071 2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 21:59:53.116431 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111074 2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 21:59:53.116431 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111077 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 21:59:53.116431 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111080 2555 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 21:59:53.116431 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111083 2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 21:59:53.116966 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111086 2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 21:59:53.116966 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111088 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 21:59:53.116966 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111091 2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 21:59:53.116966 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111098 2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 21:59:53.116966 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111101 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 21:59:53.116966 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111104 2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 21:59:53.116966 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111106 2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 21:59:53.116966 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111108 2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 21:59:53.116966 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111111 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 21:59:53.116966 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111113 2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 21:59:53.116966 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111115 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 21:59:53.116966 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111118 2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 21:59:53.116966 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111120 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 21:59:53.116966 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111122 2555 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 21:59:53.116966 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111125 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 21:59:53.116966 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111127 2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 21:59:53.116966 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111129 2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 21:59:53.116966 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111132 2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 21:59:53.116966 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111134 2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 21:59:53.116966 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111136 2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 21:59:53.117438 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111139 2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 21:59:53.117438 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111141 2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 21:59:53.117438 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111144 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 21:59:53.117438 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111146 2555 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 21:59:53.117438 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111149 2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 21:59:53.117438 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111151 2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 21:59:53.117438 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111154 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 21:59:53.117438 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111156 2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 21:59:53.117438 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111158 2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 21:59:53.117438 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111161 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 21:59:53.117438 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111163 2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 21:59:53.117438 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111167 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 21:59:53.117438 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111169 2555 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 21:59:53.117438 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111172 2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 21:59:53.117438 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111175 2555 feature_gate.go:328] unrecognized feature gate: Example Jul 13 21:59:53.117438 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111177 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 21:59:53.117438 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111185 2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 21:59:53.117438 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111188 2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 21:59:53.117438 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111190 2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 21:59:53.117438 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111192 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 21:59:53.117951 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:53.111195 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 21:59:53.117951 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.111199 2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 13 21:59:53.117951 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.111315 2555 server.go:962] "Client rotation is on, will bootstrap in background" Jul 13 21:59:53.117951 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.113380 2555 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 13 21:59:53.117951 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.115076 2555 server.go:1019] "Starting client certificate rotation" Jul 13 21:59:53.117951 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.115178 2555 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 13 21:59:53.117951 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.115209 2555 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 13 21:59:53.117951 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.117544 2555 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 13 21:59:53.122161 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.122140 2555 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 13 21:59:53.139270 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.139251 2555 log.go:25] "Validated CRI v1 runtime API" Jul 13 21:59:53.141375 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.141360 2555 log.go:25] "Validated CRI v1 image API" Jul 13 21:59:53.143541 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.143521 2555 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 13 21:59:53.147736 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.147721 2555 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 13 21:59:53.150502 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.150477 2555 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8dc6b632-8f1e-4d16-927a-73573d292489:/dev/nvme0n1p4 a8533978-81e8-45fd-813c-793c5d2aa479:/dev/nvme0n1p3] Jul 13 21:59:53.150549 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.150502 2555 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 13 21:59:53.155754 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.155644 2555 manager.go:216] Machine: {Timestamp:2026-07-13 21:59:53.15427693 +0000 UTC m=+0.378455940 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097988 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c09ff0f2b663e7ceaa8a444ec500b SystemUUID:ec2c09ff-0f2b-663e-7cea-a8a444ec500b BootID:3eeb3d21-4919-4e16-a867-f84a18c0cf68 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:c4:9e:9c:a0:fb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:c4:9e:9c:a0:fb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:e6:34:fe:f9:26:f1 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 13 21:59:53.155754 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.155752 2555 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 13 21:59:53.155868 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.155838 2555 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 13 21:59:53.158688 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.158663 2555 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 13 21:59:53.158839 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.158691 2555 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-234.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 13 21:59:53.159534 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.159524 2555 topology_manager.go:138] "Creating topology manager with none policy" Jul 13 21:59:53.159564 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.159536 2555 container_manager_linux.go:309] "Creating device plugin manager" Jul 13 21:59:53.159564 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.159544 2555 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 13 21:59:53.160913 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.160903 2555 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 13 21:59:53.162401 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.162392 2555 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 13 21:59:53.166281 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.166270 2555 state_mem.go:36] "Initialized new in-memory state store" Jul 13 21:59:53.166393 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.166383 2555 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 13 21:59:53.166425 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.166413 2555 kubelet.go:486] "Attempting to sync node with API server" Jul 13 21:59:53.166425 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.166423 2555 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 13 21:59:53.166482 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.166438 2555 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 13 21:59:53.166482 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.166447 2555 kubelet.go:398] "Adding apiserver pod source" Jul 13 21:59:53.166482 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.166455 2555 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 13 21:59:53.166871 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.166857 2555 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 13 21:59:53.166920 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.166886 2555 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 13 21:59:53.169291 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.169277 2555 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 13 21:59:53.171120 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.171108 2555 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 13 21:59:53.171161 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.171138 2555 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 13 21:59:53.171297 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.171288 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 13 21:59:53.171332 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.171302 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 13 21:59:53.171332 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.171308 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 13 21:59:53.171332 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.171313 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 13 21:59:53.171332 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.171321 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 13 21:59:53.171332 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.171326 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 13 21:59:53.171949 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.171939 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 13 21:59:53.171982 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.171950 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 13 21:59:53.171982 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.171957 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 13 21:59:53.171982 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.171962 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 13 21:59:53.171982 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.171970 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 13 21:59:53.171982 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.171978 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 13 21:59:53.172857 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.172847 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 13 21:59:53.172894 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.172858 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 13 21:59:53.172924 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.172894 2555 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ff8mf" Jul 13 21:59:53.176408 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.176394 2555 server.go:1268] "Started kubelet" Jul 13 21:59:53.176567 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.176477 2555 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 13 21:59:53.177382 ip-10-0-133-234 systemd[1]: Started Kubernetes Kubelet. Jul 13 21:59:53.180349 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.176544 2555 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 13 21:59:53.180420 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.180405 2555 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 13 21:59:53.181008 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.180984 2555 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-234.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 13 21:59:53.181115 ip-10-0-133-234 kubenswrapper[2555]: E0713 21:59:53.181028 2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-234.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 13 21:59:53.181297 ip-10-0-133-234 kubenswrapper[2555]: E0713 21:59:53.181266 2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 13 21:59:53.181593 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.181564 2555 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ff8mf" Jul 13 21:59:53.182324 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.182304 2555 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 13 21:59:53.182835 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.182818 2555 server.go:310] "Adding debug handlers to kubelet server" Jul 13 21:59:53.183332 ip-10-0-133-234 kubenswrapper[2555]: E0713 21:59:53.183297 2555 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 13 21:59:53.187403 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.187386 2555 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 13 21:59:53.187403 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.187400 2555 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 13 21:59:53.187542 ip-10-0-133-234 kubenswrapper[2555]: E0713 21:59:53.186450 2555 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-234.ec2.internal.18c1f85a13ae65b6 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-234.ec2.internal,UID:ip-10-0-133-234.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-234.ec2.internal,},FirstTimestamp:2026-07-13 21:59:53.176368566 +0000 UTC m=+0.400547563,LastTimestamp:2026-07-13 21:59:53.176368566 +0000 UTC m=+0.400547563,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-234.ec2.internal,}" Jul 13 21:59:53.187542 ip-10-0-133-234 kubenswrapper[2555]: E0713 21:59:53.187446 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-234.ec2.internal\" not found" Jul 13 21:59:53.188076 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.188057 2555 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 13 21:59:53.188173 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.188094 2555 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 13 21:59:53.188173 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.188108 2555 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 13 21:59:53.188173 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.188148 2555 factory.go:55] Registering systemd factory Jul 13 21:59:53.188173 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.188166 2555 factory.go:223] Registration of the systemd container factory successfully Jul 13 21:59:53.188365 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.188222 2555 reconstruct.go:97] "Volume reconstruction finished" Jul 13 21:59:53.188365 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.188230 2555 reconciler.go:29] "Reconciler: start to sync state" Jul 13 21:59:53.188454 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.188400 2555 factory.go:153] Registering CRI-O factory Jul 13 21:59:53.188454 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.188412 2555 factory.go:223] Registration of the crio container factory successfully Jul 13 21:59:53.188454 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.188450 2555 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 13 21:59:53.188585 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.188474 2555 factory.go:103] Registering Raw factory Jul 13 21:59:53.188585 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.188487 2555 manager.go:1196] Started watching for new ooms in manager Jul 13 21:59:53.189132 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.189117 2555 manager.go:349] Starting recovery of all containers Jul 13 21:59:53.190623 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.190607 2555 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 13 21:59:53.196275 ip-10-0-133-234 kubenswrapper[2555]: E0713 21:59:53.196252 2555 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-133-234.ec2.internal\" not found" node="ip-10-0-133-234.ec2.internal" Jul 13 21:59:53.199091 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.198938 2555 manager.go:354] Recovery completed Jul 13 21:59:53.204809 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.204795 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 21:59:53.208302 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.208286 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-234.ec2.internal" event="NodeHasSufficientMemory" Jul 13 21:59:53.208362 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.208317 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-234.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 21:59:53.208362 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.208327 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-234.ec2.internal" event="NodeHasSufficientPID" Jul 13 21:59:53.208936 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.208920 2555 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 13 21:59:53.208936 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.208933 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 21:59:53.208936 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.208934 2555 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 13 21:59:53.209111 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.208954 2555 state_mem.go:36] "Initialized new in-memory state store" Jul 13 21:59:53.210906 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.210891 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-234.ec2.internal" event="NodeHasSufficientMemory" Jul 13 21:59:53.210976 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.210922 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-234.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 21:59:53.210976 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.210937 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-234.ec2.internal" event="NodeHasSufficientPID" Jul 13 21:59:53.211151 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.211097 2555 policy_none.go:49] "None policy: Start" Jul 13 21:59:53.211151 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.211112 2555 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 13 21:59:53.211151 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.211122 2555 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 13 21:59:53.213106 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.213095 2555 policy_none.go:47] "Start" Jul 13 21:59:53.248368 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.248346 2555 manager.go:337] "Starting Device Plugin manager" Jul 13 21:59:53.248456 ip-10-0-133-234 kubenswrapper[2555]: E0713 21:59:53.248382 2555 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 13 21:59:53.248456 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.248397 2555 server.go:87] "Starting device plugin registration server" Jul 13 21:59:53.248652 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.248640 2555 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 13 21:59:53.248711 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.248654 2555 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 13 21:59:53.248755 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.248740 2555 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 13 21:59:53.248944 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.248831 2555 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 13 21:59:53.248944 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.248846 2555 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 13 21:59:53.249419 ip-10-0-133-234 kubenswrapper[2555]: E0713 21:59:53.249401 2555 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 13 21:59:53.249480 ip-10-0-133-234 kubenswrapper[2555]: E0713 21:59:53.249455 2555 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-234.ec2.internal\" not found" Jul 13 21:59:53.321141 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.321106 2555 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 13 21:59:53.323210 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.322438 2555 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 13 21:59:53.323210 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.322474 2555 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 13 21:59:53.323210 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.322497 2555 kubelet.go:2448] "Starting kubelet main sync loop" Jul 13 21:59:53.323210 ip-10-0-133-234 kubenswrapper[2555]: E0713 21:59:53.322537 2555 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 13 21:59:53.324811 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.324793 2555 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 13 21:59:53.349498 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.349456 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 21:59:53.350310 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.350296 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-234.ec2.internal" event="NodeHasSufficientMemory" Jul 13 21:59:53.350377 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.350326 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-234.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 21:59:53.350377 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.350338 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-234.ec2.internal" event="NodeHasSufficientPID" Jul 13 21:59:53.350377 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.350361 2555 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-234.ec2.internal" Jul 13 21:59:53.358020 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.358007 2555 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-234.ec2.internal" Jul 13 21:59:53.358068 ip-10-0-133-234 kubenswrapper[2555]: E0713 21:59:53.358026 2555 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-234.ec2.internal\": node \"ip-10-0-133-234.ec2.internal\" not found" Jul 13 21:59:53.374328 ip-10-0-133-234 kubenswrapper[2555]: E0713 21:59:53.374311 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-234.ec2.internal\" not found" Jul 13 21:59:53.423545 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.423512 2555 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-234.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-234.ec2.internal"] Jul 13 21:59:53.423647 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.423590 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 21:59:53.425222 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.425207 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-234.ec2.internal" event="NodeHasSufficientMemory" Jul 13 21:59:53.425320 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.425240 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-234.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 21:59:53.425320 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.425255 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-234.ec2.internal" event="NodeHasSufficientPID" Jul 13 21:59:53.427475 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.427460 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 21:59:53.427573 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.427563 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-234.ec2.internal" Jul 13 21:59:53.427623 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.427595 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 21:59:53.428525 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.428507 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-234.ec2.internal" event="NodeHasSufficientMemory" Jul 13 21:59:53.428628 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.428536 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-234.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 21:59:53.428628 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.428549 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-234.ec2.internal" event="NodeHasSufficientPID" Jul 13 21:59:53.428628 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.428560 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-234.ec2.internal" event="NodeHasSufficientMemory" Jul 13 21:59:53.428628 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.428582 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-234.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 21:59:53.428628 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.428594 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-234.ec2.internal" event="NodeHasSufficientPID" Jul 13 21:59:53.430733 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.430718 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-234.ec2.internal" Jul 13 21:59:53.430815 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.430745 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 21:59:53.431691 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.431667 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-234.ec2.internal" event="NodeHasSufficientMemory" Jul 13 21:59:53.431760 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.431696 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-234.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 21:59:53.431760 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.431710 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-234.ec2.internal" event="NodeHasSufficientPID" Jul 13 21:59:53.454369 ip-10-0-133-234 kubenswrapper[2555]: E0713 21:59:53.454348 2555 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-234.ec2.internal\" not found" node="ip-10-0-133-234.ec2.internal" Jul 13 21:59:53.458541 ip-10-0-133-234 kubenswrapper[2555]: E0713 21:59:53.458527 2555 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-234.ec2.internal\" not found" node="ip-10-0-133-234.ec2.internal" Jul 13 21:59:53.474406 ip-10-0-133-234 kubenswrapper[2555]: E0713 21:59:53.474385 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-234.ec2.internal\" not found" Jul 13 21:59:53.489852 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.489829 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6fba7a483df990721f423a08a6cb4680-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-234.ec2.internal\" (UID: \"6fba7a483df990721f423a08a6cb4680\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-234.ec2.internal" Jul 13 21:59:53.489910 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.489860 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2df761e96899302463c4ca9d4dd2ee01-config\") pod \"kube-apiserver-proxy-ip-10-0-133-234.ec2.internal\" (UID: \"2df761e96899302463c4ca9d4dd2ee01\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-234.ec2.internal" Jul 13 21:59:53.489910 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.489878 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6fba7a483df990721f423a08a6cb4680-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-234.ec2.internal\" (UID: \"6fba7a483df990721f423a08a6cb4680\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-234.ec2.internal" Jul 13 21:59:53.574677 ip-10-0-133-234 kubenswrapper[2555]: E0713 21:59:53.574651 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-234.ec2.internal\" not found" Jul 13 21:59:53.591031 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.591007 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6fba7a483df990721f423a08a6cb4680-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-234.ec2.internal\" (UID: \"6fba7a483df990721f423a08a6cb4680\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-234.ec2.internal" Jul 13 21:59:53.591095 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.591038 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6fba7a483df990721f423a08a6cb4680-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-234.ec2.internal\" (UID: \"6fba7a483df990721f423a08a6cb4680\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-234.ec2.internal" Jul 13 21:59:53.591095 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.591062 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2df761e96899302463c4ca9d4dd2ee01-config\") pod \"kube-apiserver-proxy-ip-10-0-133-234.ec2.internal\" (UID: \"2df761e96899302463c4ca9d4dd2ee01\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-234.ec2.internal" Jul 13 21:59:53.591095 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.591075 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6fba7a483df990721f423a08a6cb4680-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-234.ec2.internal\" (UID: \"6fba7a483df990721f423a08a6cb4680\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-234.ec2.internal" Jul 13 21:59:53.591204 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.591111 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2df761e96899302463c4ca9d4dd2ee01-config\") pod \"kube-apiserver-proxy-ip-10-0-133-234.ec2.internal\" (UID: \"2df761e96899302463c4ca9d4dd2ee01\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-234.ec2.internal" Jul 13 21:59:53.591204 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.591112 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6fba7a483df990721f423a08a6cb4680-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-234.ec2.internal\" (UID: \"6fba7a483df990721f423a08a6cb4680\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-234.ec2.internal" Jul 13 21:59:53.675739 ip-10-0-133-234 kubenswrapper[2555]: E0713 21:59:53.675655 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-234.ec2.internal\" not found" Jul 13 21:59:53.757220 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.757187 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-234.ec2.internal" Jul 13 21:59:53.760960 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:53.760940 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-234.ec2.internal" Jul 13 21:59:53.776576 ip-10-0-133-234 kubenswrapper[2555]: E0713 21:59:53.776556 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-234.ec2.internal\" not found" Jul 13 21:59:55.107517 ip-10-0-133-234 kubenswrapper[2555]: E0713 21:59:55.107494 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-234.ec2.internal\" not found" Jul 13 21:59:55.208019 ip-10-0-133-234 kubenswrapper[2555]: E0713 21:59:55.207954 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-234.ec2.internal\" not found" Jul 13 21:59:55.308453 ip-10-0-133-234 kubenswrapper[2555]: E0713 21:59:55.308423 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-234.ec2.internal\" not found" Jul 13 21:59:55.344880 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:55.344853 2555 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 13 21:59:55.345364 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:55.344988 2555 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 13 21:59:55.345364 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:55.345026 2555 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 13 21:59:55.409278 ip-10-0-133-234 kubenswrapper[2555]: E0713 21:59:55.409256 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-234.ec2.internal\" not found" Jul 13 21:59:55.418697 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:55.418671 2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-12 21:54:54 +0000 UTC" deadline="2028-02-02 09:55:49.320433712 +0000 UTC" Jul 13 21:59:55.418697 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:55.418697 2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13643h55m53.901739964s" Jul 13 21:59:55.498371 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:55.498304 2555 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 13 21:59:55.510013 ip-10-0-133-234 kubenswrapper[2555]: E0713 21:59:55.509985 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-234.ec2.internal\" not found" Jul 13 21:59:55.584071 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:55.584034 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fba7a483df990721f423a08a6cb4680.slice/crio-f1b844ce4a0b6f3e52b540a2baeb2bb40c13eb555b35f63016834236f6a4feff WatchSource:0}: Error finding container f1b844ce4a0b6f3e52b540a2baeb2bb40c13eb555b35f63016834236f6a4feff: Status 404 returned error can't find the container with id f1b844ce4a0b6f3e52b540a2baeb2bb40c13eb555b35f63016834236f6a4feff Jul 13 21:59:55.584304 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:55.584284 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2df761e96899302463c4ca9d4dd2ee01.slice/crio-985f36c6284e2c5f57922fdf7726889cd5579e54f98d10732e3ffb68397e77dc WatchSource:0}: Error finding container 985f36c6284e2c5f57922fdf7726889cd5579e54f98d10732e3ffb68397e77dc: Status 404 returned error can't find the container with id 985f36c6284e2c5f57922fdf7726889cd5579e54f98d10732e3ffb68397e77dc Jul 13 21:59:55.589817 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:55.589794 2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 13 21:59:55.610989 ip-10-0-133-234 kubenswrapper[2555]: E0713 21:59:55.610958 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-234.ec2.internal\" not found" Jul 13 21:59:55.711476 ip-10-0-133-234 kubenswrapper[2555]: E0713 21:59:55.711444 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-234.ec2.internal\" not found" Jul 13 21:59:55.811978 ip-10-0-133-234 kubenswrapper[2555]: E0713 21:59:55.811902 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-234.ec2.internal\" not found" Jul 13 21:59:55.912654 ip-10-0-133-234 kubenswrapper[2555]: E0713 21:59:55.912625 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-234.ec2.internal\" not found" Jul 13 21:59:56.011914 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.011882 2555 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 13 21:59:56.018415 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.018390 2555 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-234.ec2.internal" Jul 13 21:59:56.034962 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.034847 2555 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 13 21:59:56.036216 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.036175 2555 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-234.ec2.internal" Jul 13 21:59:56.048594 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.048513 2555 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 13 21:59:56.398474 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.398432 2555 apiserver.go:52] "Watching apiserver" Jul 13 21:59:56.406106 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.405923 2555 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 13 21:59:56.407373 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.407337 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-87w2d","kube-system/konnectivity-agent-q8965","openshift-cluster-node-tuning-operator/tuned-kr77b","openshift-multus/multus-86hvh","openshift-multus/multus-additional-cni-plugins-6xnxl","openshift-network-diagnostics/network-check-target-rg8tn","openshift-network-operator/iptables-alerter-zs8wl","openshift-ovn-kubernetes/ovnkube-node-tbknb","kube-system/global-pull-secret-syncer-5bkm9","kube-system/kube-apiserver-proxy-ip-10-0-133-234.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vt7w","openshift-dns/node-resolver-2znr7","openshift-image-registry/node-ca-6nq95","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-234.ec2.internal"] Jul 13 21:59:56.410065 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.410039 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zs8wl" Jul 13 21:59:56.412222 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.412183 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6nq95" Jul 13 21:59:56.413108 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.413086 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 13 21:59:56.413363 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.413342 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 13 21:59:56.413464 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.413363 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 13 21:59:56.413464 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.413374 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hx8pb\"" Jul 13 21:59:56.414251 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.414231 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kr77b" Jul 13 21:59:56.414341 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.414329 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 13 21:59:56.414654 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.414633 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 13 21:59:56.414654 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.414650 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pxknk\"" Jul 13 21:59:56.414757 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.414687 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 13 21:59:56.416370 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.416352 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 13 21:59:56.416466 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.416356 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-b976d\"" Jul 13 21:59:56.416525 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.416486 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-86hvh" Jul 13 21:59:56.416888 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.416652 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 13 21:59:56.419349 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.419186 2555 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 13 21:59:56.422419 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.420454 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-2xbj7\"" Jul 13 21:59:56.422419 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.420689 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 13 21:59:56.422419 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.420869 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 13 21:59:56.422419 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.421063 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 13 21:59:56.422419 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.421278 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 13 21:59:56.422419 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.421576 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6xnxl" Jul 13 21:59:56.424012 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.423842 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 13 21:59:56.424104 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.424005 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 13 21:59:56.424159 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.424113 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jl6dz\"" Jul 13 21:59:56.428095 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.427837 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rg8tn" Jul 13 21:59:56.428095 ip-10-0-133-234 kubenswrapper[2555]: E0713 21:59:56.427913 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rg8tn" podUID="d1b9ac92-b63a-4180-9293-86c6b0c545a0" Jul 13 21:59:56.430102 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.430076 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5bkm9" Jul 13 21:59:56.430186 ip-10-0-133-234 kubenswrapper[2555]: E0713 21:59:56.430161 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5bkm9" podUID="56086743-e831-415a-a2c3-6982ef099751" Jul 13 21:59:56.430308 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.430289 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/888489a9-c409-4275-8318-ffee8ce2a187-tmp\") pod \"tuned-kr77b\" (UID: \"888489a9-c409-4275-8318-ffee8ce2a187\") " pod="openshift-cluster-node-tuning-operator/tuned-kr77b" Jul 13 21:59:56.430385 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.430319 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/22a82d33-c9c8-445a-a814-5534be9dd95e-host-var-lib-cni-bin\") pod \"multus-86hvh\" (UID: \"22a82d33-c9c8-445a-a814-5534be9dd95e\") " pod="openshift-multus/multus-86hvh" Jul 13 21:59:56.430385 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.430342 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xszk\" (UniqueName: \"kubernetes.io/projected/22a82d33-c9c8-445a-a814-5534be9dd95e-kube-api-access-4xszk\") pod \"multus-86hvh\" (UID: \"22a82d33-c9c8-445a-a814-5534be9dd95e\") " pod="openshift-multus/multus-86hvh" Jul 13 21:59:56.430385 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.430358 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ttzk\" (UniqueName: \"kubernetes.io/projected/d94f40ee-a429-4bf2-b7cb-c436922cdedc-kube-api-access-4ttzk\") pod \"multus-additional-cni-plugins-6xnxl\" (UID: \"d94f40ee-a429-4bf2-b7cb-c436922cdedc\") " pod="openshift-multus/multus-additional-cni-plugins-6xnxl" Jul 13 21:59:56.430385 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.430381 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/888489a9-c409-4275-8318-ffee8ce2a187-host\") pod \"tuned-kr77b\" (UID: \"888489a9-c409-4275-8318-ffee8ce2a187\") " pod="openshift-cluster-node-tuning-operator/tuned-kr77b" Jul 13 21:59:56.430576 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.430405 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fj6l\" (UniqueName: \"kubernetes.io/projected/888489a9-c409-4275-8318-ffee8ce2a187-kube-api-access-6fj6l\") pod \"tuned-kr77b\" (UID: \"888489a9-c409-4275-8318-ffee8ce2a187\") " pod="openshift-cluster-node-tuning-operator/tuned-kr77b" Jul 13 21:59:56.430576 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.430430 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/22a82d33-c9c8-445a-a814-5534be9dd95e-host-run-netns\") pod \"multus-86hvh\" (UID: \"22a82d33-c9c8-445a-a814-5534be9dd95e\") " pod="openshift-multus/multus-86hvh" Jul 13 21:59:56.430576 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.430473 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/22a82d33-c9c8-445a-a814-5534be9dd95e-hostroot\") pod \"multus-86hvh\" (UID: \"22a82d33-c9c8-445a-a814-5534be9dd95e\") " pod="openshift-multus/multus-86hvh" Jul 13 21:59:56.430576 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.430514 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d94f40ee-a429-4bf2-b7cb-c436922cdedc-cni-binary-copy\") pod \"multus-additional-cni-plugins-6xnxl\" (UID: \"d94f40ee-a429-4bf2-b7cb-c436922cdedc\") " pod="openshift-multus/multus-additional-cni-plugins-6xnxl" Jul 13 21:59:56.430751 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.430594 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a8651358-951f-4485-b3d8-9e21c26b7fc4-iptables-alerter-script\") pod \"iptables-alerter-zs8wl\" (UID: \"a8651358-951f-4485-b3d8-9e21c26b7fc4\") " pod="openshift-network-operator/iptables-alerter-zs8wl" Jul 13 21:59:56.430751 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.430622 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/54550ef2-07f8-4e03-b898-c5d7536cc61e-serviceca\") pod \"node-ca-6nq95\" (UID: \"54550ef2-07f8-4e03-b898-c5d7536cc61e\") " pod="openshift-image-registry/node-ca-6nq95" Jul 13 21:59:56.430751 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.430639 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/888489a9-c409-4275-8318-ffee8ce2a187-etc-modprobe-d\") pod \"tuned-kr77b\" (UID: \"888489a9-c409-4275-8318-ffee8ce2a187\") " pod="openshift-cluster-node-tuning-operator/tuned-kr77b" Jul 13 21:59:56.430751 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.430658 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/888489a9-c409-4275-8318-ffee8ce2a187-lib-modules\") pod \"tuned-kr77b\" (UID: \"888489a9-c409-4275-8318-ffee8ce2a187\") " pod="openshift-cluster-node-tuning-operator/tuned-kr77b" Jul 13 21:59:56.430751 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.430672 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/22a82d33-c9c8-445a-a814-5534be9dd95e-multus-cni-dir\") pod \"multus-86hvh\" (UID: \"22a82d33-c9c8-445a-a814-5534be9dd95e\") " pod="openshift-multus/multus-86hvh" Jul 13 21:59:56.430751 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.430685 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/22a82d33-c9c8-445a-a814-5534be9dd95e-cni-binary-copy\") pod \"multus-86hvh\" (UID: \"22a82d33-c9c8-445a-a814-5534be9dd95e\") " pod="openshift-multus/multus-86hvh" Jul 13 21:59:56.430751 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.430722 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/22a82d33-c9c8-445a-a814-5534be9dd95e-host-var-lib-cni-multus\") pod \"multus-86hvh\" (UID: \"22a82d33-c9c8-445a-a814-5534be9dd95e\") " pod="openshift-multus/multus-86hvh" Jul 13 21:59:56.431096 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.431028 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/22a82d33-c9c8-445a-a814-5534be9dd95e-host-var-lib-kubelet\") pod \"multus-86hvh\" (UID: \"22a82d33-c9c8-445a-a814-5534be9dd95e\") " pod="openshift-multus/multus-86hvh" Jul 13 21:59:56.431096 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.431062 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b6bs\" (UniqueName: \"kubernetes.io/projected/a8651358-951f-4485-b3d8-9e21c26b7fc4-kube-api-access-4b6bs\") pod \"iptables-alerter-zs8wl\" (UID: \"a8651358-951f-4485-b3d8-9e21c26b7fc4\") " pod="openshift-network-operator/iptables-alerter-zs8wl" Jul 13 21:59:56.431220 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.431096 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/888489a9-c409-4275-8318-ffee8ce2a187-etc-sysctl-conf\") pod \"tuned-kr77b\" (UID: \"888489a9-c409-4275-8318-ffee8ce2a187\") " pod="openshift-cluster-node-tuning-operator/tuned-kr77b" Jul 13 21:59:56.431220 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.431121 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/22a82d33-c9c8-445a-a814-5534be9dd95e-multus-conf-dir\") pod \"multus-86hvh\" (UID: \"22a82d33-c9c8-445a-a814-5534be9dd95e\") " pod="openshift-multus/multus-86hvh" Jul 13 21:59:56.431220 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.431145 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22a82d33-c9c8-445a-a814-5534be9dd95e-etc-kubernetes\") pod \"multus-86hvh\" (UID: \"22a82d33-c9c8-445a-a814-5534be9dd95e\") " pod="openshift-multus/multus-86hvh" Jul 13 21:59:56.431220 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.431165 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a8651358-951f-4485-b3d8-9e21c26b7fc4-host-slash\") pod \"iptables-alerter-zs8wl\" (UID: \"a8651358-951f-4485-b3d8-9e21c26b7fc4\") " pod="openshift-network-operator/iptables-alerter-zs8wl" Jul 13 21:59:56.431220 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.431189 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/888489a9-c409-4275-8318-ffee8ce2a187-etc-systemd\") pod \"tuned-kr77b\" (UID: \"888489a9-c409-4275-8318-ffee8ce2a187\") " pod="openshift-cluster-node-tuning-operator/tuned-kr77b" Jul 13 21:59:56.431453 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.431232 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/888489a9-c409-4275-8318-ffee8ce2a187-var-lib-kubelet\") pod \"tuned-kr77b\" (UID: \"888489a9-c409-4275-8318-ffee8ce2a187\") " pod="openshift-cluster-node-tuning-operator/tuned-kr77b" Jul 13 21:59:56.431453 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.431256 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/22a82d33-c9c8-445a-a814-5534be9dd95e-system-cni-dir\") pod \"multus-86hvh\" (UID: \"22a82d33-c9c8-445a-a814-5534be9dd95e\") " pod="openshift-multus/multus-86hvh" Jul 13 21:59:56.431453 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.431280 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/22a82d33-c9c8-445a-a814-5534be9dd95e-multus-socket-dir-parent\") pod \"multus-86hvh\" (UID: \"22a82d33-c9c8-445a-a814-5534be9dd95e\") " pod="openshift-multus/multus-86hvh" Jul 13 21:59:56.431453 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.431303 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d94f40ee-a429-4bf2-b7cb-c436922cdedc-cnibin\") pod \"multus-additional-cni-plugins-6xnxl\" (UID: \"d94f40ee-a429-4bf2-b7cb-c436922cdedc\") " pod="openshift-multus/multus-additional-cni-plugins-6xnxl" Jul 13 21:59:56.431453 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.431324 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/888489a9-c409-4275-8318-ffee8ce2a187-etc-tuned\") pod \"tuned-kr77b\" (UID: \"888489a9-c409-4275-8318-ffee8ce2a187\") " pod="openshift-cluster-node-tuning-operator/tuned-kr77b" Jul 13 21:59:56.431453 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.431350 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/22a82d33-c9c8-445a-a814-5534be9dd95e-os-release\") pod \"multus-86hvh\" (UID: \"22a82d33-c9c8-445a-a814-5534be9dd95e\") " pod="openshift-multus/multus-86hvh" Jul 13 21:59:56.431453 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.431377 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/22a82d33-c9c8-445a-a814-5534be9dd95e-multus-daemon-config\") pod \"multus-86hvh\" (UID: \"22a82d33-c9c8-445a-a814-5534be9dd95e\") " pod="openshift-multus/multus-86hvh" Jul 13 21:59:56.431453 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.431402 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d94f40ee-a429-4bf2-b7cb-c436922cdedc-os-release\") pod \"multus-additional-cni-plugins-6xnxl\" (UID: \"d94f40ee-a429-4bf2-b7cb-c436922cdedc\") " pod="openshift-multus/multus-additional-cni-plugins-6xnxl" Jul 13 21:59:56.431814 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.431453 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/54550ef2-07f8-4e03-b898-c5d7536cc61e-host\") pod \"node-ca-6nq95\" (UID: \"54550ef2-07f8-4e03-b898-c5d7536cc61e\") " pod="openshift-image-registry/node-ca-6nq95" Jul 13 21:59:56.431814 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.431480 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/888489a9-c409-4275-8318-ffee8ce2a187-run\") pod \"tuned-kr77b\" (UID: \"888489a9-c409-4275-8318-ffee8ce2a187\") " pod="openshift-cluster-node-tuning-operator/tuned-kr77b" Jul 13 21:59:56.431814 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.431503 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/22a82d33-c9c8-445a-a814-5534be9dd95e-cnibin\") pod \"multus-86hvh\" (UID: \"22a82d33-c9c8-445a-a814-5534be9dd95e\") " pod="openshift-multus/multus-86hvh" Jul 13 21:59:56.431814 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.431533 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/22a82d33-c9c8-445a-a814-5534be9dd95e-host-run-k8s-cni-cncf-io\") pod \"multus-86hvh\" (UID: \"22a82d33-c9c8-445a-a814-5534be9dd95e\") " pod="openshift-multus/multus-86hvh" Jul 13 21:59:56.431814 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.431573 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/22a82d33-c9c8-445a-a814-5534be9dd95e-host-run-multus-certs\") pod \"multus-86hvh\" (UID: \"22a82d33-c9c8-445a-a814-5534be9dd95e\") " pod="openshift-multus/multus-86hvh" Jul 13 21:59:56.431814 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.431604 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d94f40ee-a429-4bf2-b7cb-c436922cdedc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6xnxl\" (UID: \"d94f40ee-a429-4bf2-b7cb-c436922cdedc\") " pod="openshift-multus/multus-additional-cni-plugins-6xnxl" Jul 13 21:59:56.431814 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.431644 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/888489a9-c409-4275-8318-ffee8ce2a187-etc-sysconfig\") pod \"tuned-kr77b\" (UID: \"888489a9-c409-4275-8318-ffee8ce2a187\") " pod="openshift-cluster-node-tuning-operator/tuned-kr77b" Jul 13 21:59:56.431814 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.431680 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d94f40ee-a429-4bf2-b7cb-c436922cdedc-system-cni-dir\") pod \"multus-additional-cni-plugins-6xnxl\" (UID: \"d94f40ee-a429-4bf2-b7cb-c436922cdedc\") " pod="openshift-multus/multus-additional-cni-plugins-6xnxl" Jul 13 21:59:56.431814 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.431703 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d94f40ee-a429-4bf2-b7cb-c436922cdedc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6xnxl\" (UID: \"d94f40ee-a429-4bf2-b7cb-c436922cdedc\") " pod="openshift-multus/multus-additional-cni-plugins-6xnxl" Jul 13 21:59:56.431814 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.431752 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d94f40ee-a429-4bf2-b7cb-c436922cdedc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6xnxl\" (UID: \"d94f40ee-a429-4bf2-b7cb-c436922cdedc\") " pod="openshift-multus/multus-additional-cni-plugins-6xnxl" Jul 13 21:59:56.431814 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.431779 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kqsp\" (UniqueName: \"kubernetes.io/projected/54550ef2-07f8-4e03-b898-c5d7536cc61e-kube-api-access-2kqsp\") pod \"node-ca-6nq95\" (UID: \"54550ef2-07f8-4e03-b898-c5d7536cc61e\") " pod="openshift-image-registry/node-ca-6nq95" Jul 13 21:59:56.431814 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.431801 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/888489a9-c409-4275-8318-ffee8ce2a187-etc-kubernetes\") pod \"tuned-kr77b\" (UID: \"888489a9-c409-4275-8318-ffee8ce2a187\") " pod="openshift-cluster-node-tuning-operator/tuned-kr77b" Jul 13 21:59:56.432222 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.431823 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/888489a9-c409-4275-8318-ffee8ce2a187-etc-sysctl-d\") pod \"tuned-kr77b\" (UID: \"888489a9-c409-4275-8318-ffee8ce2a187\") " pod="openshift-cluster-node-tuning-operator/tuned-kr77b" Jul 13 21:59:56.432222 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.431848 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/888489a9-c409-4275-8318-ffee8ce2a187-sys\") pod \"tuned-kr77b\" (UID: \"888489a9-c409-4275-8318-ffee8ce2a187\") " pod="openshift-cluster-node-tuning-operator/tuned-kr77b" Jul 13 21:59:56.433401 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.433382 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87w2d" Jul 13 21:59:56.433510 ip-10-0-133-234 kubenswrapper[2555]: E0713 21:59:56.433486 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87w2d" podUID="55ac7566-430e-4e45-ab69-fe943cfb4c44" Jul 13 21:59:56.435881 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.435864 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-q8965" Jul 13 21:59:56.438048 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.438022 2555 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 13 21:59:56.438285 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.438270 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 13 21:59:56.438889 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.438873 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 13 21:59:56.439231 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.439184 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-v44dd\"" Jul 13 21:59:56.439798 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.439780 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" Jul 13 21:59:56.442388 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.442015 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vt7w" Jul 13 21:59:56.445086 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.445064 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2znr7" Jul 13 21:59:56.445337 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.445319 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 13 21:59:56.445552 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.445538 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-nmw8g\"" Jul 13 21:59:56.445611 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.445585 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 13 21:59:56.445702 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.445690 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 13 21:59:56.445757 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.445719 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 13 21:59:56.446232 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.446035 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 13 21:59:56.446232 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.446079 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 13 21:59:56.446763 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.446562 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 13 21:59:56.446763 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.446585 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 13 21:59:56.446763 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.446638 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-w85hd\"" Jul 13 21:59:56.446763 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.446585 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 13 21:59:56.447186 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.447167 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-s4gsv\"" Jul 13 21:59:56.447421 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.447364 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 13 21:59:56.447687 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.447667 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 13 21:59:56.458168 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.458146 2555 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-fx9nv" Jul 13 21:59:56.466982 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.466964 2555 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-fx9nv" Jul 13 21:59:56.520105 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.520081 2555 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 13 21:59:56.532155 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.532131 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/888489a9-c409-4275-8318-ffee8ce2a187-host\") pod \"tuned-kr77b\" (UID: \"888489a9-c409-4275-8318-ffee8ce2a187\") " pod="openshift-cluster-node-tuning-operator/tuned-kr77b" Jul 13 21:59:56.532293 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.532159 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/22a82d33-c9c8-445a-a814-5534be9dd95e-host-run-netns\") pod \"multus-86hvh\" (UID: \"22a82d33-c9c8-445a-a814-5534be9dd95e\") " pod="openshift-multus/multus-86hvh" Jul 13 21:59:56.532293 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.532184 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/22a82d33-c9c8-445a-a814-5534be9dd95e-hostroot\") pod \"multus-86hvh\" (UID: \"22a82d33-c9c8-445a-a814-5534be9dd95e\") " pod="openshift-multus/multus-86hvh" Jul 13 21:59:56.532293 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.532215 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d94f40ee-a429-4bf2-b7cb-c436922cdedc-cni-binary-copy\") pod \"multus-additional-cni-plugins-6xnxl\" (UID: \"d94f40ee-a429-4bf2-b7cb-c436922cdedc\") " pod="openshift-multus/multus-additional-cni-plugins-6xnxl" Jul 13 21:59:56.532293 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.532275 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/22a82d33-c9c8-445a-a814-5534be9dd95e-host-run-netns\") pod \"multus-86hvh\" (UID: \"22a82d33-c9c8-445a-a814-5534be9dd95e\") " pod="openshift-multus/multus-86hvh" Jul 13 21:59:56.532486 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.532312 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8h4k\" (UniqueName: \"kubernetes.io/projected/d1b9ac92-b63a-4180-9293-86c6b0c545a0-kube-api-access-h8h4k\") pod \"network-check-target-rg8tn\" (UID: \"d1b9ac92-b63a-4180-9293-86c6b0c545a0\") " pod="openshift-network-diagnostics/network-check-target-rg8tn" Jul 13 21:59:56.532486 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.532339 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56086743-e831-415a-a2c3-6982ef099751-original-pull-secret\") pod \"global-pull-secret-syncer-5bkm9\" (UID: \"56086743-e831-415a-a2c3-6982ef099751\") " pod="kube-system/global-pull-secret-syncer-5bkm9" Jul 13 21:59:56.532486 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.532360 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e2bb252b-36a7-4723-b3bd-e3eb873e7bed-run-openvswitch\") pod \"ovnkube-node-tbknb\" (UID: \"e2bb252b-36a7-4723-b3bd-e3eb873e7bed\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" Jul 13 21:59:56.532486 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.532385 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a8651358-951f-4485-b3d8-9e21c26b7fc4-iptables-alerter-script\") pod \"iptables-alerter-zs8wl\" (UID: \"a8651358-951f-4485-b3d8-9e21c26b7fc4\") " pod="openshift-network-operator/iptables-alerter-zs8wl" Jul 13 21:59:56.532486 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.532409 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/888489a9-c409-4275-8318-ffee8ce2a187-etc-modprobe-d\") pod \"tuned-kr77b\" (UID: \"888489a9-c409-4275-8318-ffee8ce2a187\") " pod="openshift-cluster-node-tuning-operator/tuned-kr77b" Jul 13 21:59:56.532486 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.532433 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/888489a9-c409-4275-8318-ffee8ce2a187-lib-modules\") pod \"tuned-kr77b\" (UID: \"888489a9-c409-4275-8318-ffee8ce2a187\") " pod="openshift-cluster-node-tuning-operator/tuned-kr77b" Jul 13 21:59:56.532486 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.532434 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/888489a9-c409-4275-8318-ffee8ce2a187-host\") pod \"tuned-kr77b\" (UID: \"888489a9-c409-4275-8318-ffee8ce2a187\") " pod="openshift-cluster-node-tuning-operator/tuned-kr77b" Jul 13 21:59:56.532486 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.532456 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/22a82d33-c9c8-445a-a814-5534be9dd95e-multus-cni-dir\") pod \"multus-86hvh\" (UID: \"22a82d33-c9c8-445a-a814-5534be9dd95e\") " pod="openshift-multus/multus-86hvh" Jul 13 21:59:56.532486 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.532480 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/22a82d33-c9c8-445a-a814-5534be9dd95e-cni-binary-copy\") pod \"multus-86hvh\" (UID: \"22a82d33-c9c8-445a-a814-5534be9dd95e\") " pod="openshift-multus/multus-86hvh" Jul 13 21:59:56.532901 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.532486 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/22a82d33-c9c8-445a-a814-5534be9dd95e-hostroot\") pod \"multus-86hvh\" (UID: \"22a82d33-c9c8-445a-a814-5534be9dd95e\") " pod="openshift-multus/multus-86hvh" Jul 13 21:59:56.532901 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.532507 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/22a82d33-c9c8-445a-a814-5534be9dd95e-host-var-lib-cni-multus\") pod \"multus-86hvh\" (UID: \"22a82d33-c9c8-445a-a814-5534be9dd95e\") " pod="openshift-multus/multus-86hvh" Jul 13 21:59:56.532901 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.532668 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/22a82d33-c9c8-445a-a814-5534be9dd95e-multus-conf-dir\") pod \"multus-86hvh\" (UID: \"22a82d33-c9c8-445a-a814-5534be9dd95e\") " pod="openshift-multus/multus-86hvh" Jul 13 21:59:56.532901 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.532826 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7kjm\" (UniqueName: \"kubernetes.io/projected/cf29609f-4a54-4c17-a7ca-10a0bdab7d28-kube-api-access-m7kjm\") pod \"aws-ebs-csi-driver-node-9vt7w\" (UID: \"cf29609f-4a54-4c17-a7ca-10a0bdab7d28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vt7w" Jul 13 21:59:56.532901 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.532870 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b6bs\" (UniqueName: \"kubernetes.io/projected/a8651358-951f-4485-b3d8-9e21c26b7fc4-kube-api-access-4b6bs\") pod \"iptables-alerter-zs8wl\" (UID: \"a8651358-951f-4485-b3d8-9e21c26b7fc4\") " pod="openshift-network-operator/iptables-alerter-zs8wl" Jul 13 21:59:56.533133 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.532903 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/888489a9-c409-4275-8318-ffee8ce2a187-etc-sysctl-conf\") pod \"tuned-kr77b\" (UID: \"888489a9-c409-4275-8318-ffee8ce2a187\") " pod="openshift-cluster-node-tuning-operator/tuned-kr77b" Jul 13 21:59:56.533133 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.532945 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lzdd\" (UniqueName: \"kubernetes.io/projected/9985a1e9-5e38-4bea-9745-988c4524fc21-kube-api-access-4lzdd\") pod \"node-resolver-2znr7\" (UID: \"9985a1e9-5e38-4bea-9745-988c4524fc21\") " pod="openshift-dns/node-resolver-2znr7" Jul 13 21:59:56.533133 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.532937 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/888489a9-c409-4275-8318-ffee8ce2a187-etc-modprobe-d\") pod \"tuned-kr77b\" (UID: \"888489a9-c409-4275-8318-ffee8ce2a187\") " pod="openshift-cluster-node-tuning-operator/tuned-kr77b" Jul 13 21:59:56.533133 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.532979 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/56086743-e831-415a-a2c3-6982ef099751-kubelet-config\") pod \"global-pull-secret-syncer-5bkm9\" (UID: \"56086743-e831-415a-a2c3-6982ef099751\") " pod="kube-system/global-pull-secret-syncer-5bkm9" Jul 13 21:59:56.533133 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.533034 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/22a82d33-c9c8-445a-a814-5534be9dd95e-multus-conf-dir\") pod \"multus-86hvh\" (UID: \"22a82d33-c9c8-445a-a814-5534be9dd95e\") " pod="openshift-multus/multus-86hvh" Jul 13 21:59:56.533133 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.533077 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e2bb252b-36a7-4723-b3bd-e3eb873e7bed-run-systemd\") pod \"ovnkube-node-tbknb\" (UID: \"e2bb252b-36a7-4723-b3bd-e3eb873e7bed\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" Jul 13 21:59:56.533436 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.533148 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d94f40ee-a429-4bf2-b7cb-c436922cdedc-cni-binary-copy\") pod \"multus-additional-cni-plugins-6xnxl\" (UID: \"d94f40ee-a429-4bf2-b7cb-c436922cdedc\") " pod="openshift-multus/multus-additional-cni-plugins-6xnxl" Jul 13 21:59:56.533436 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.533176 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/888489a9-c409-4275-8318-ffee8ce2a187-etc-sysctl-conf\") pod \"tuned-kr77b\" (UID: \"888489a9-c409-4275-8318-ffee8ce2a187\") " pod="openshift-cluster-node-tuning-operator/tuned-kr77b" Jul 13 21:59:56.533436 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.533276 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/22a82d33-c9c8-445a-a814-5534be9dd95e-host-var-lib-cni-multus\") pod \"multus-86hvh\" (UID: \"22a82d33-c9c8-445a-a814-5534be9dd95e\") " pod="openshift-multus/multus-86hvh" Jul 13 21:59:56.533436 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.533327 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/888489a9-c409-4275-8318-ffee8ce2a187-lib-modules\") pod \"tuned-kr77b\" (UID: \"888489a9-c409-4275-8318-ffee8ce2a187\") " pod="openshift-cluster-node-tuning-operator/tuned-kr77b" Jul 13 21:59:56.533436 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.533360 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/22a82d33-c9c8-445a-a814-5534be9dd95e-multus-cni-dir\") pod \"multus-86hvh\" (UID: \"22a82d33-c9c8-445a-a814-5534be9dd95e\") " pod="openshift-multus/multus-86hvh" Jul 13 21:59:56.533717 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.533688 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e2bb252b-36a7-4723-b3bd-e3eb873e7bed-run-ovn\") pod \"ovnkube-node-tbknb\" (UID: \"e2bb252b-36a7-4723-b3bd-e3eb873e7bed\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" Jul 13 21:59:56.533825 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.533734 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e2bb252b-36a7-4723-b3bd-e3eb873e7bed-host-cni-bin\") pod \"ovnkube-node-tbknb\" (UID: \"e2bb252b-36a7-4723-b3bd-e3eb873e7bed\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" Jul 13 21:59:56.533825 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.533773 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/888489a9-c409-4275-8318-ffee8ce2a187-etc-systemd\") pod \"tuned-kr77b\" (UID: \"888489a9-c409-4275-8318-ffee8ce2a187\") " pod="openshift-cluster-node-tuning-operator/tuned-kr77b" Jul 13 21:59:56.533825 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.533805 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/22a82d33-c9c8-445a-a814-5534be9dd95e-system-cni-dir\") pod \"multus-86hvh\" (UID: \"22a82d33-c9c8-445a-a814-5534be9dd95e\") " pod="openshift-multus/multus-86hvh" Jul 13 21:59:56.533973 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.533836 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/22a82d33-c9c8-445a-a814-5534be9dd95e-multus-socket-dir-parent\") pod \"multus-86hvh\" (UID: \"22a82d33-c9c8-445a-a814-5534be9dd95e\") " pod="openshift-multus/multus-86hvh" Jul 13 21:59:56.533973 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.533883 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a8651358-951f-4485-b3d8-9e21c26b7fc4-iptables-alerter-script\") pod \"iptables-alerter-zs8wl\" (UID: \"a8651358-951f-4485-b3d8-9e21c26b7fc4\") " pod="openshift-network-operator/iptables-alerter-zs8wl" Jul 13 21:59:56.533973 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.533905 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/22a82d33-c9c8-445a-a814-5534be9dd95e-cni-binary-copy\") pod \"multus-86hvh\" (UID: \"22a82d33-c9c8-445a-a814-5534be9dd95e\") " pod="openshift-multus/multus-86hvh" Jul 13 21:59:56.533973 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.533907 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d94f40ee-a429-4bf2-b7cb-c436922cdedc-cnibin\") pod \"multus-additional-cni-plugins-6xnxl\" (UID: \"d94f40ee-a429-4bf2-b7cb-c436922cdedc\") " pod="openshift-multus/multus-additional-cni-plugins-6xnxl" Jul 13 21:59:56.533973 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.533947 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtbf5\" (UniqueName: \"kubernetes.io/projected/55ac7566-430e-4e45-ab69-fe943cfb4c44-kube-api-access-vtbf5\") pod \"network-metrics-daemon-87w2d\" (UID: \"55ac7566-430e-4e45-ab69-fe943cfb4c44\") " pod="openshift-multus/network-metrics-daemon-87w2d" Jul 13 21:59:56.534194 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.533977 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/22a82d33-c9c8-445a-a814-5534be9dd95e-multus-socket-dir-parent\") pod \"multus-86hvh\" (UID: \"22a82d33-c9c8-445a-a814-5534be9dd95e\") " pod="openshift-multus/multus-86hvh" Jul 13 21:59:56.534194 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.533987 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cf29609f-4a54-4c17-a7ca-10a0bdab7d28-socket-dir\") pod \"aws-ebs-csi-driver-node-9vt7w\" (UID: \"cf29609f-4a54-4c17-a7ca-10a0bdab7d28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vt7w" Jul 13 21:59:56.534194 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.533991 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d94f40ee-a429-4bf2-b7cb-c436922cdedc-cnibin\") pod \"multus-additional-cni-plugins-6xnxl\" (UID: \"d94f40ee-a429-4bf2-b7cb-c436922cdedc\") " pod="openshift-multus/multus-additional-cni-plugins-6xnxl" Jul 13 21:59:56.534194 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.534042 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e2bb252b-36a7-4723-b3bd-e3eb873e7bed-ovnkube-script-lib\") pod \"ovnkube-node-tbknb\" (UID: \"e2bb252b-36a7-4723-b3bd-e3eb873e7bed\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" Jul 13 21:59:56.534383 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.534239 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/888489a9-c409-4275-8318-ffee8ce2a187-etc-tuned\") pod \"tuned-kr77b\" (UID: \"888489a9-c409-4275-8318-ffee8ce2a187\") " pod="openshift-cluster-node-tuning-operator/tuned-kr77b" Jul 13 21:59:56.534383 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.534271 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/22a82d33-c9c8-445a-a814-5534be9dd95e-os-release\") pod \"multus-86hvh\" (UID: \"22a82d33-c9c8-445a-a814-5534be9dd95e\") " pod="openshift-multus/multus-86hvh" Jul 13 21:59:56.534383 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.534338 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/22a82d33-c9c8-445a-a814-5534be9dd95e-multus-daemon-config\") pod \"multus-86hvh\" (UID: \"22a82d33-c9c8-445a-a814-5534be9dd95e\") " pod="openshift-multus/multus-86hvh" Jul 13 21:59:56.534383 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.534369 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55ac7566-430e-4e45-ab69-fe943cfb4c44-metrics-certs\") pod \"network-metrics-daemon-87w2d\" (UID: \"55ac7566-430e-4e45-ab69-fe943cfb4c44\") " pod="openshift-multus/network-metrics-daemon-87w2d" Jul 13 21:59:56.534554 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.534435 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cf29609f-4a54-4c17-a7ca-10a0bdab7d28-etc-selinux\") pod \"aws-ebs-csi-driver-node-9vt7w\" (UID: \"cf29609f-4a54-4c17-a7ca-10a0bdab7d28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vt7w" Jul 13 21:59:56.534554 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.534491 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e2bb252b-36a7-4723-b3bd-e3eb873e7bed-host-slash\") pod \"ovnkube-node-tbknb\" (UID: \"e2bb252b-36a7-4723-b3bd-e3eb873e7bed\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" Jul 13 21:59:56.534642 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.534553 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e2bb252b-36a7-4723-b3bd-e3eb873e7bed-node-log\") pod \"ovnkube-node-tbknb\" (UID: \"e2bb252b-36a7-4723-b3bd-e3eb873e7bed\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" Jul 13 21:59:56.534642 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.534582 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e2bb252b-36a7-4723-b3bd-e3eb873e7bed-env-overrides\") pod \"ovnkube-node-tbknb\" (UID: \"e2bb252b-36a7-4723-b3bd-e3eb873e7bed\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" Jul 13 21:59:56.534737 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.534676 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/22a82d33-c9c8-445a-a814-5534be9dd95e-host-run-k8s-cni-cncf-io\") pod \"multus-86hvh\" (UID: \"22a82d33-c9c8-445a-a814-5534be9dd95e\") " pod="openshift-multus/multus-86hvh" Jul 13 21:59:56.534737 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.534706 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d94f40ee-a429-4bf2-b7cb-c436922cdedc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6xnxl\" (UID: \"d94f40ee-a429-4bf2-b7cb-c436922cdedc\") " pod="openshift-multus/multus-additional-cni-plugins-6xnxl" Jul 13 21:59:56.534827 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.534751 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cf29609f-4a54-4c17-a7ca-10a0bdab7d28-sys-fs\") pod \"aws-ebs-csi-driver-node-9vt7w\" (UID: \"cf29609f-4a54-4c17-a7ca-10a0bdab7d28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vt7w" Jul 13 21:59:56.534827 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.534778 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e2bb252b-36a7-4723-b3bd-e3eb873e7bed-host-run-netns\") pod \"ovnkube-node-tbknb\" (UID: \"e2bb252b-36a7-4723-b3bd-e3eb873e7bed\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" Jul 13 21:59:56.534827 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.534819 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e2bb252b-36a7-4723-b3bd-e3eb873e7bed-etc-openvswitch\") pod \"ovnkube-node-tbknb\" (UID: \"e2bb252b-36a7-4723-b3bd-e3eb873e7bed\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" Jul 13 21:59:56.535044 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.535023 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/888489a9-c409-4275-8318-ffee8ce2a187-sys\") pod \"tuned-kr77b\" (UID: \"888489a9-c409-4275-8318-ffee8ce2a187\") " pod="openshift-cluster-node-tuning-operator/tuned-kr77b" Jul 13 21:59:56.535154 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.535056 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/888489a9-c409-4275-8318-ffee8ce2a187-tmp\") pod \"tuned-kr77b\" (UID: \"888489a9-c409-4275-8318-ffee8ce2a187\") " pod="openshift-cluster-node-tuning-operator/tuned-kr77b" Jul 13 21:59:56.535154 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.535106 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/22a82d33-c9c8-445a-a814-5534be9dd95e-host-var-lib-cni-bin\") pod \"multus-86hvh\" (UID: \"22a82d33-c9c8-445a-a814-5534be9dd95e\") " pod="openshift-multus/multus-86hvh" Jul 13 21:59:56.535364 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.535163 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xszk\" (UniqueName: \"kubernetes.io/projected/22a82d33-c9c8-445a-a814-5534be9dd95e-kube-api-access-4xszk\") pod \"multus-86hvh\" (UID: \"22a82d33-c9c8-445a-a814-5534be9dd95e\") " pod="openshift-multus/multus-86hvh" Jul 13 21:59:56.535364 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.535225 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/56086743-e831-415a-a2c3-6982ef099751-dbus\") pod \"global-pull-secret-syncer-5bkm9\" (UID: \"56086743-e831-415a-a2c3-6982ef099751\") " pod="kube-system/global-pull-secret-syncer-5bkm9" Jul 13 21:59:56.535364 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.535249 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e2bb252b-36a7-4723-b3bd-e3eb873e7bed-host-cni-netd\") pod \"ovnkube-node-tbknb\" (UID: \"e2bb252b-36a7-4723-b3bd-e3eb873e7bed\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" Jul 13 21:59:56.535364 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.535308 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fj6l\" (UniqueName: \"kubernetes.io/projected/888489a9-c409-4275-8318-ffee8ce2a187-kube-api-access-6fj6l\") pod \"tuned-kr77b\" (UID: \"888489a9-c409-4275-8318-ffee8ce2a187\") " pod="openshift-cluster-node-tuning-operator/tuned-kr77b" Jul 13 21:59:56.535364 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.535337 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cf29609f-4a54-4c17-a7ca-10a0bdab7d28-device-dir\") pod \"aws-ebs-csi-driver-node-9vt7w\" (UID: \"cf29609f-4a54-4c17-a7ca-10a0bdab7d28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vt7w" Jul 13 21:59:56.535676 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.535364 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2bb252b-36a7-4723-b3bd-e3eb873e7bed-host-run-ovn-kubernetes\") pod \"ovnkube-node-tbknb\" (UID: \"e2bb252b-36a7-4723-b3bd-e3eb873e7bed\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" Jul 13 21:59:56.535676 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.535389 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/54550ef2-07f8-4e03-b898-c5d7536cc61e-serviceca\") pod \"node-ca-6nq95\" (UID: \"54550ef2-07f8-4e03-b898-c5d7536cc61e\") " pod="openshift-image-registry/node-ca-6nq95" Jul 13 21:59:56.535676 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.535419 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/22a82d33-c9c8-445a-a814-5534be9dd95e-host-var-lib-kubelet\") pod \"multus-86hvh\" (UID: \"22a82d33-c9c8-445a-a814-5534be9dd95e\") " pod="openshift-multus/multus-86hvh" Jul 13 21:59:56.535676 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.535489 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22a82d33-c9c8-445a-a814-5534be9dd95e-etc-kubernetes\") pod \"multus-86hvh\" (UID: \"22a82d33-c9c8-445a-a814-5534be9dd95e\") " pod="openshift-multus/multus-86hvh" Jul 13 21:59:56.535676 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.535519 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e2bb252b-36a7-4723-b3bd-e3eb873e7bed-var-lib-openvswitch\") pod \"ovnkube-node-tbknb\" (UID: \"e2bb252b-36a7-4723-b3bd-e3eb873e7bed\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" Jul 13 21:59:56.535676 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.535566 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csg7d\" (UniqueName: \"kubernetes.io/projected/e2bb252b-36a7-4723-b3bd-e3eb873e7bed-kube-api-access-csg7d\") pod \"ovnkube-node-tbknb\" (UID: \"e2bb252b-36a7-4723-b3bd-e3eb873e7bed\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" Jul 13 21:59:56.536007 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.535988 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e2bb252b-36a7-4723-b3bd-e3eb873e7bed-systemd-units\") pod \"ovnkube-node-tbknb\" (UID: \"e2bb252b-36a7-4723-b3bd-e3eb873e7bed\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" Jul 13 21:59:56.536099 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.536054 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a8651358-951f-4485-b3d8-9e21c26b7fc4-host-slash\") pod \"iptables-alerter-zs8wl\" (UID: \"a8651358-951f-4485-b3d8-9e21c26b7fc4\") " pod="openshift-network-operator/iptables-alerter-zs8wl" Jul 13 21:59:56.536099 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.536084 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/888489a9-c409-4275-8318-ffee8ce2a187-var-lib-kubelet\") pod \"tuned-kr77b\" (UID: \"888489a9-c409-4275-8318-ffee8ce2a187\") " pod="openshift-cluster-node-tuning-operator/tuned-kr77b" Jul 13 21:59:56.536193 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.536113 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2bb252b-36a7-4723-b3bd-e3eb873e7bed-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tbknb\" (UID: \"e2bb252b-36a7-4723-b3bd-e3eb873e7bed\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" Jul 13 21:59:56.536269 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.536227 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d94f40ee-a429-4bf2-b7cb-c436922cdedc-os-release\") pod \"multus-additional-cni-plugins-6xnxl\" (UID: \"d94f40ee-a429-4bf2-b7cb-c436922cdedc\") " pod="openshift-multus/multus-additional-cni-plugins-6xnxl" Jul 13 21:59:56.536269 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.536257 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cf29609f-4a54-4c17-a7ca-10a0bdab7d28-registration-dir\") pod \"aws-ebs-csi-driver-node-9vt7w\" (UID: \"cf29609f-4a54-4c17-a7ca-10a0bdab7d28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vt7w" Jul 13 21:59:56.536353 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.536284 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9985a1e9-5e38-4bea-9745-988c4524fc21-hosts-file\") pod \"node-resolver-2znr7\" (UID: \"9985a1e9-5e38-4bea-9745-988c4524fc21\") " pod="openshift-dns/node-resolver-2znr7" Jul 13 21:59:56.536353 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.536337 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/061cea64-4ced-48b8-bd3a-3c6c22fc9e77-agent-certs\") pod \"konnectivity-agent-q8965\" (UID: \"061cea64-4ced-48b8-bd3a-3c6c22fc9e77\") " pod="kube-system/konnectivity-agent-q8965" Jul 13 21:59:56.536435 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.536379 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2bb252b-36a7-4723-b3bd-e3eb873e7bed-host-kubelet\") pod \"ovnkube-node-tbknb\" (UID: \"e2bb252b-36a7-4723-b3bd-e3eb873e7bed\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" Jul 13 21:59:56.536435 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.536403 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/54550ef2-07f8-4e03-b898-c5d7536cc61e-host\") pod \"node-ca-6nq95\" (UID: \"54550ef2-07f8-4e03-b898-c5d7536cc61e\") " pod="openshift-image-registry/node-ca-6nq95" Jul 13 21:59:56.536435 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.536429 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/888489a9-c409-4275-8318-ffee8ce2a187-run\") pod \"tuned-kr77b\" (UID: \"888489a9-c409-4275-8318-ffee8ce2a187\") " pod="openshift-cluster-node-tuning-operator/tuned-kr77b" Jul 13 21:59:56.536568 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.536478 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/22a82d33-c9c8-445a-a814-5534be9dd95e-system-cni-dir\") pod \"multus-86hvh\" (UID: \"22a82d33-c9c8-445a-a814-5534be9dd95e\") " pod="openshift-multus/multus-86hvh" Jul 13 21:59:56.536568 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.536496 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/22a82d33-c9c8-445a-a814-5534be9dd95e-cnibin\") pod \"multus-86hvh\" (UID: \"22a82d33-c9c8-445a-a814-5534be9dd95e\") " pod="openshift-multus/multus-86hvh" Jul 13 21:59:56.536568 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.536501 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/22a82d33-c9c8-445a-a814-5534be9dd95e-host-var-lib-kubelet\") pod \"multus-86hvh\" (UID: \"22a82d33-c9c8-445a-a814-5534be9dd95e\") " pod="openshift-multus/multus-86hvh" Jul 13 21:59:56.536568 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.536536 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a8651358-951f-4485-b3d8-9e21c26b7fc4-host-slash\") pod \"iptables-alerter-zs8wl\" (UID: \"a8651358-951f-4485-b3d8-9e21c26b7fc4\") " pod="openshift-network-operator/iptables-alerter-zs8wl" Jul 13 21:59:56.536752 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.536564 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/22a82d33-c9c8-445a-a814-5534be9dd95e-host-run-multus-certs\") pod \"multus-86hvh\" (UID: \"22a82d33-c9c8-445a-a814-5534be9dd95e\") " pod="openshift-multus/multus-86hvh" Jul 13 21:59:56.536752 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.536600 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/888489a9-c409-4275-8318-ffee8ce2a187-var-lib-kubelet\") pod \"tuned-kr77b\" (UID: \"888489a9-c409-4275-8318-ffee8ce2a187\") " pod="openshift-cluster-node-tuning-operator/tuned-kr77b" Jul 13 21:59:56.536752 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.536608 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d94f40ee-a429-4bf2-b7cb-c436922cdedc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6xnxl\" (UID: \"d94f40ee-a429-4bf2-b7cb-c436922cdedc\") " pod="openshift-multus/multus-additional-cni-plugins-6xnxl" Jul 13 21:59:56.536752 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.536614 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/22a82d33-c9c8-445a-a814-5534be9dd95e-cnibin\") pod \"multus-86hvh\" (UID: \"22a82d33-c9c8-445a-a814-5534be9dd95e\") " pod="openshift-multus/multus-86hvh" Jul 13 21:59:56.536752 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.536662 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/22a82d33-c9c8-445a-a814-5534be9dd95e-host-run-multus-certs\") pod \"multus-86hvh\" (UID: \"22a82d33-c9c8-445a-a814-5534be9dd95e\") " pod="openshift-multus/multus-86hvh" Jul 13 21:59:56.536752 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.536678 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/54550ef2-07f8-4e03-b898-c5d7536cc61e-host\") pod \"node-ca-6nq95\" (UID: \"54550ef2-07f8-4e03-b898-c5d7536cc61e\") " pod="openshift-image-registry/node-ca-6nq95" Jul 13 21:59:56.536752 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.536690 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9985a1e9-5e38-4bea-9745-988c4524fc21-tmp-dir\") pod \"node-resolver-2znr7\" (UID: \"9985a1e9-5e38-4bea-9745-988c4524fc21\") " pod="openshift-dns/node-resolver-2znr7" Jul 13 21:59:56.536752 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.536693 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d94f40ee-a429-4bf2-b7cb-c436922cdedc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6xnxl\" (UID: \"d94f40ee-a429-4bf2-b7cb-c436922cdedc\") " pod="openshift-multus/multus-additional-cni-plugins-6xnxl" Jul 13 21:59:56.536752 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.536728 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/061cea64-4ced-48b8-bd3a-3c6c22fc9e77-konnectivity-ca\") pod \"konnectivity-agent-q8965\" (UID: \"061cea64-4ced-48b8-bd3a-3c6c22fc9e77\") " pod="kube-system/konnectivity-agent-q8965" Jul 13 21:59:56.536752 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.536730 2555 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 13 21:59:56.537226 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.536733 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/888489a9-c409-4275-8318-ffee8ce2a187-run\") pod \"tuned-kr77b\" (UID: \"888489a9-c409-4275-8318-ffee8ce2a187\") " pod="openshift-cluster-node-tuning-operator/tuned-kr77b" Jul 13 21:59:56.537226 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.536798 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d94f40ee-a429-4bf2-b7cb-c436922cdedc-os-release\") pod \"multus-additional-cni-plugins-6xnxl\" (UID: \"d94f40ee-a429-4bf2-b7cb-c436922cdedc\") " pod="openshift-multus/multus-additional-cni-plugins-6xnxl" Jul 13 21:59:56.537226 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.536839 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/22a82d33-c9c8-445a-a814-5534be9dd95e-host-var-lib-cni-bin\") pod \"multus-86hvh\" (UID: \"22a82d33-c9c8-445a-a814-5534be9dd95e\") " pod="openshift-multus/multus-86hvh" Jul 13 21:59:56.537226 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.536845 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/22a82d33-c9c8-445a-a814-5534be9dd95e-host-run-k8s-cni-cncf-io\") pod \"multus-86hvh\" (UID: \"22a82d33-c9c8-445a-a814-5534be9dd95e\") " pod="openshift-multus/multus-86hvh" Jul 13 21:59:56.537226 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.536891 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e2bb252b-36a7-4723-b3bd-e3eb873e7bed-log-socket\") pod \"ovnkube-node-tbknb\" (UID: \"e2bb252b-36a7-4723-b3bd-e3eb873e7bed\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" Jul 13 21:59:56.537226 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.536937 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/888489a9-c409-4275-8318-ffee8ce2a187-etc-sysconfig\") pod \"tuned-kr77b\" (UID: \"888489a9-c409-4275-8318-ffee8ce2a187\") " pod="openshift-cluster-node-tuning-operator/tuned-kr77b" Jul 13 21:59:56.537226 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.536972 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d94f40ee-a429-4bf2-b7cb-c436922cdedc-system-cni-dir\") pod \"multus-additional-cni-plugins-6xnxl\" (UID: \"d94f40ee-a429-4bf2-b7cb-c436922cdedc\") " pod="openshift-multus/multus-additional-cni-plugins-6xnxl" Jul 13 21:59:56.537226 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.537024 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d94f40ee-a429-4bf2-b7cb-c436922cdedc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6xnxl\" (UID: \"d94f40ee-a429-4bf2-b7cb-c436922cdedc\") " pod="openshift-multus/multus-additional-cni-plugins-6xnxl" Jul 13 21:59:56.537226 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.537058 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cf29609f-4a54-4c17-a7ca-10a0bdab7d28-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9vt7w\" (UID: \"cf29609f-4a54-4c17-a7ca-10a0bdab7d28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vt7w" Jul 13 21:59:56.537226 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.537083 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e2bb252b-36a7-4723-b3bd-e3eb873e7bed-ovn-node-metrics-cert\") pod \"ovnkube-node-tbknb\" (UID: \"e2bb252b-36a7-4723-b3bd-e3eb873e7bed\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" Jul 13 21:59:56.537226 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.537115 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kqsp\" (UniqueName: \"kubernetes.io/projected/54550ef2-07f8-4e03-b898-c5d7536cc61e-kube-api-access-2kqsp\") pod \"node-ca-6nq95\" (UID: \"54550ef2-07f8-4e03-b898-c5d7536cc61e\") " pod="openshift-image-registry/node-ca-6nq95" Jul 13 21:59:56.537226 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.537156 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d94f40ee-a429-4bf2-b7cb-c436922cdedc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6xnxl\" (UID: \"d94f40ee-a429-4bf2-b7cb-c436922cdedc\") " pod="openshift-multus/multus-additional-cni-plugins-6xnxl" Jul 13 21:59:56.537226 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.537175 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/54550ef2-07f8-4e03-b898-c5d7536cc61e-serviceca\") pod \"node-ca-6nq95\" (UID: \"54550ef2-07f8-4e03-b898-c5d7536cc61e\") " pod="openshift-image-registry/node-ca-6nq95" Jul 13 21:59:56.537810 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.537235 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/888489a9-c409-4275-8318-ffee8ce2a187-etc-systemd\") pod \"tuned-kr77b\" (UID: \"888489a9-c409-4275-8318-ffee8ce2a187\") " pod="openshift-cluster-node-tuning-operator/tuned-kr77b" Jul 13 21:59:56.537810 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.537251 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22a82d33-c9c8-445a-a814-5534be9dd95e-etc-kubernetes\") pod \"multus-86hvh\" (UID: \"22a82d33-c9c8-445a-a814-5534be9dd95e\") " pod="openshift-multus/multus-86hvh" Jul 13 21:59:56.537810 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.537364 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/888489a9-c409-4275-8318-ffee8ce2a187-etc-kubernetes\") pod \"tuned-kr77b\" (UID: \"888489a9-c409-4275-8318-ffee8ce2a187\") " pod="openshift-cluster-node-tuning-operator/tuned-kr77b" Jul 13 21:59:56.537810 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.537404 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/888489a9-c409-4275-8318-ffee8ce2a187-etc-sysctl-d\") pod \"tuned-kr77b\" (UID: \"888489a9-c409-4275-8318-ffee8ce2a187\") " pod="openshift-cluster-node-tuning-operator/tuned-kr77b" Jul 13 21:59:56.537810 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.537439 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ttzk\" (UniqueName: \"kubernetes.io/projected/d94f40ee-a429-4bf2-b7cb-c436922cdedc-kube-api-access-4ttzk\") pod \"multus-additional-cni-plugins-6xnxl\" (UID: \"d94f40ee-a429-4bf2-b7cb-c436922cdedc\") " pod="openshift-multus/multus-additional-cni-plugins-6xnxl" Jul 13 21:59:56.537810 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.537503 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e2bb252b-36a7-4723-b3bd-e3eb873e7bed-ovnkube-config\") pod \"ovnkube-node-tbknb\" (UID: \"e2bb252b-36a7-4723-b3bd-e3eb873e7bed\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" Jul 13 21:59:56.537810 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.537535 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/888489a9-c409-4275-8318-ffee8ce2a187-sys\") pod \"tuned-kr77b\" (UID: \"888489a9-c409-4275-8318-ffee8ce2a187\") " pod="openshift-cluster-node-tuning-operator/tuned-kr77b" Jul 13 21:59:56.537810 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.537607 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d94f40ee-a429-4bf2-b7cb-c436922cdedc-system-cni-dir\") pod \"multus-additional-cni-plugins-6xnxl\" (UID: \"d94f40ee-a429-4bf2-b7cb-c436922cdedc\") " pod="openshift-multus/multus-additional-cni-plugins-6xnxl" Jul 13 21:59:56.538223 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.537802 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/888489a9-c409-4275-8318-ffee8ce2a187-etc-sysconfig\") pod \"tuned-kr77b\" (UID: \"888489a9-c409-4275-8318-ffee8ce2a187\") " pod="openshift-cluster-node-tuning-operator/tuned-kr77b" Jul 13 21:59:56.538223 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.537993 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/22a82d33-c9c8-445a-a814-5534be9dd95e-os-release\") pod \"multus-86hvh\" (UID: \"22a82d33-c9c8-445a-a814-5534be9dd95e\") " pod="openshift-multus/multus-86hvh" Jul 13 21:59:56.538223 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.538019 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/888489a9-c409-4275-8318-ffee8ce2a187-etc-kubernetes\") pod \"tuned-kr77b\" (UID: \"888489a9-c409-4275-8318-ffee8ce2a187\") " pod="openshift-cluster-node-tuning-operator/tuned-kr77b" Jul 13 21:59:56.538618 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.538600 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d94f40ee-a429-4bf2-b7cb-c436922cdedc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6xnxl\" (UID: \"d94f40ee-a429-4bf2-b7cb-c436922cdedc\") " pod="openshift-multus/multus-additional-cni-plugins-6xnxl" Jul 13 21:59:56.538679 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.538659 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/888489a9-c409-4275-8318-ffee8ce2a187-etc-sysctl-d\") pod \"tuned-kr77b\" (UID: \"888489a9-c409-4275-8318-ffee8ce2a187\") " pod="openshift-cluster-node-tuning-operator/tuned-kr77b" Jul 13 21:59:56.538840 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.538825 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/22a82d33-c9c8-445a-a814-5534be9dd95e-multus-daemon-config\") pod \"multus-86hvh\" (UID: \"22a82d33-c9c8-445a-a814-5534be9dd95e\") " pod="openshift-multus/multus-86hvh" Jul 13 21:59:56.540522 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.540501 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/888489a9-c409-4275-8318-ffee8ce2a187-tmp\") pod \"tuned-kr77b\" (UID: \"888489a9-c409-4275-8318-ffee8ce2a187\") " pod="openshift-cluster-node-tuning-operator/tuned-kr77b" Jul 13 21:59:56.541253 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.541233 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/888489a9-c409-4275-8318-ffee8ce2a187-etc-tuned\") pod \"tuned-kr77b\" (UID: \"888489a9-c409-4275-8318-ffee8ce2a187\") " pod="openshift-cluster-node-tuning-operator/tuned-kr77b" Jul 13 21:59:56.548769 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.548562 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kqsp\" (UniqueName: \"kubernetes.io/projected/54550ef2-07f8-4e03-b898-c5d7536cc61e-kube-api-access-2kqsp\") pod \"node-ca-6nq95\" (UID: \"54550ef2-07f8-4e03-b898-c5d7536cc61e\") " pod="openshift-image-registry/node-ca-6nq95" Jul 13 21:59:56.548769 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.548743 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xszk\" (UniqueName: \"kubernetes.io/projected/22a82d33-c9c8-445a-a814-5534be9dd95e-kube-api-access-4xszk\") pod \"multus-86hvh\" (UID: \"22a82d33-c9c8-445a-a814-5534be9dd95e\") " pod="openshift-multus/multus-86hvh" Jul 13 21:59:56.549149 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.549132 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fj6l\" (UniqueName: \"kubernetes.io/projected/888489a9-c409-4275-8318-ffee8ce2a187-kube-api-access-6fj6l\") pod \"tuned-kr77b\" (UID: \"888489a9-c409-4275-8318-ffee8ce2a187\") " pod="openshift-cluster-node-tuning-operator/tuned-kr77b" Jul 13 21:59:56.549529 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.549505 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ttzk\" (UniqueName: \"kubernetes.io/projected/d94f40ee-a429-4bf2-b7cb-c436922cdedc-kube-api-access-4ttzk\") pod \"multus-additional-cni-plugins-6xnxl\" (UID: \"d94f40ee-a429-4bf2-b7cb-c436922cdedc\") " pod="openshift-multus/multus-additional-cni-plugins-6xnxl" Jul 13 21:59:56.553023 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.552981 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b6bs\" (UniqueName: \"kubernetes.io/projected/a8651358-951f-4485-b3d8-9e21c26b7fc4-kube-api-access-4b6bs\") pod \"iptables-alerter-zs8wl\" (UID: \"a8651358-951f-4485-b3d8-9e21c26b7fc4\") " pod="openshift-network-operator/iptables-alerter-zs8wl" Jul 13 21:59:56.557721 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.557677 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-234.ec2.internal" event={"ID":"2df761e96899302463c4ca9d4dd2ee01","Type":"ContainerStarted","Data":"985f36c6284e2c5f57922fdf7726889cd5579e54f98d10732e3ffb68397e77dc"} Jul 13 21:59:56.558741 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.558719 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-234.ec2.internal" event={"ID":"6fba7a483df990721f423a08a6cb4680","Type":"ContainerStarted","Data":"f1b844ce4a0b6f3e52b540a2baeb2bb40c13eb555b35f63016834236f6a4feff"} Jul 13 21:59:56.638865 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.638822 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8h4k\" (UniqueName: \"kubernetes.io/projected/d1b9ac92-b63a-4180-9293-86c6b0c545a0-kube-api-access-h8h4k\") pod \"network-check-target-rg8tn\" (UID: \"d1b9ac92-b63a-4180-9293-86c6b0c545a0\") " pod="openshift-network-diagnostics/network-check-target-rg8tn" Jul 13 21:59:56.639013 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.638870 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56086743-e831-415a-a2c3-6982ef099751-original-pull-secret\") pod \"global-pull-secret-syncer-5bkm9\" (UID: \"56086743-e831-415a-a2c3-6982ef099751\") " pod="kube-system/global-pull-secret-syncer-5bkm9" Jul 13 21:59:56.639013 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.638888 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e2bb252b-36a7-4723-b3bd-e3eb873e7bed-run-openvswitch\") pod \"ovnkube-node-tbknb\" (UID: \"e2bb252b-36a7-4723-b3bd-e3eb873e7bed\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" Jul 13 21:59:56.639013 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.638911 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7kjm\" (UniqueName: \"kubernetes.io/projected/cf29609f-4a54-4c17-a7ca-10a0bdab7d28-kube-api-access-m7kjm\") pod \"aws-ebs-csi-driver-node-9vt7w\" (UID: \"cf29609f-4a54-4c17-a7ca-10a0bdab7d28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vt7w" Jul 13 21:59:56.639013 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.638966 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e2bb252b-36a7-4723-b3bd-e3eb873e7bed-run-openvswitch\") pod \"ovnkube-node-tbknb\" (UID: \"e2bb252b-36a7-4723-b3bd-e3eb873e7bed\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" Jul 13 21:59:56.639013 ip-10-0-133-234 kubenswrapper[2555]: E0713 21:59:56.638993 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 21:59:56.639288 ip-10-0-133-234 kubenswrapper[2555]: E0713 21:59:56.639078 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56086743-e831-415a-a2c3-6982ef099751-original-pull-secret podName:56086743-e831-415a-a2c3-6982ef099751 nodeName:}" failed. No retries permitted until 2026-07-13 21:59:57.139044843 +0000 UTC m=+3.132799131 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/56086743-e831-415a-a2c3-6982ef099751-original-pull-secret") pod "global-pull-secret-syncer-5bkm9" (UID: "56086743-e831-415a-a2c3-6982ef099751") : object "kube-system"/"original-pull-secret" not registered Jul 13 21:59:56.639288 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.639106 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lzdd\" (UniqueName: \"kubernetes.io/projected/9985a1e9-5e38-4bea-9745-988c4524fc21-kube-api-access-4lzdd\") pod \"node-resolver-2znr7\" (UID: \"9985a1e9-5e38-4bea-9745-988c4524fc21\") " pod="openshift-dns/node-resolver-2znr7" Jul 13 21:59:56.639288 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.639131 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/56086743-e831-415a-a2c3-6982ef099751-kubelet-config\") pod \"global-pull-secret-syncer-5bkm9\" (UID: \"56086743-e831-415a-a2c3-6982ef099751\") " pod="kube-system/global-pull-secret-syncer-5bkm9" Jul 13 21:59:56.639288 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.639156 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e2bb252b-36a7-4723-b3bd-e3eb873e7bed-run-systemd\") pod \"ovnkube-node-tbknb\" (UID: \"e2bb252b-36a7-4723-b3bd-e3eb873e7bed\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" Jul 13 21:59:56.639288 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.639173 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e2bb252b-36a7-4723-b3bd-e3eb873e7bed-run-ovn\") pod \"ovnkube-node-tbknb\" (UID: \"e2bb252b-36a7-4723-b3bd-e3eb873e7bed\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" Jul 13 21:59:56.639288 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.639189 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e2bb252b-36a7-4723-b3bd-e3eb873e7bed-host-cni-bin\") pod \"ovnkube-node-tbknb\" (UID: \"e2bb252b-36a7-4723-b3bd-e3eb873e7bed\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" Jul 13 21:59:56.639288 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.639254 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e2bb252b-36a7-4723-b3bd-e3eb873e7bed-run-systemd\") pod \"ovnkube-node-tbknb\" (UID: \"e2bb252b-36a7-4723-b3bd-e3eb873e7bed\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" Jul 13 21:59:56.639288 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.639257 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e2bb252b-36a7-4723-b3bd-e3eb873e7bed-host-cni-bin\") pod \"ovnkube-node-tbknb\" (UID: \"e2bb252b-36a7-4723-b3bd-e3eb873e7bed\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" Jul 13 21:59:56.639288 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.639279 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtbf5\" (UniqueName: \"kubernetes.io/projected/55ac7566-430e-4e45-ab69-fe943cfb4c44-kube-api-access-vtbf5\") pod \"network-metrics-daemon-87w2d\" (UID: \"55ac7566-430e-4e45-ab69-fe943cfb4c44\") " pod="openshift-multus/network-metrics-daemon-87w2d" Jul 13 21:59:56.639288 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.639257 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e2bb252b-36a7-4723-b3bd-e3eb873e7bed-run-ovn\") pod \"ovnkube-node-tbknb\" (UID: \"e2bb252b-36a7-4723-b3bd-e3eb873e7bed\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" Jul 13 21:59:56.639802 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.639300 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/56086743-e831-415a-a2c3-6982ef099751-kubelet-config\") pod \"global-pull-secret-syncer-5bkm9\" (UID: \"56086743-e831-415a-a2c3-6982ef099751\") " pod="kube-system/global-pull-secret-syncer-5bkm9" Jul 13 21:59:56.639802 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.639307 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cf29609f-4a54-4c17-a7ca-10a0bdab7d28-socket-dir\") pod \"aws-ebs-csi-driver-node-9vt7w\" (UID: \"cf29609f-4a54-4c17-a7ca-10a0bdab7d28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vt7w" Jul 13 21:59:56.639802 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.639373 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e2bb252b-36a7-4723-b3bd-e3eb873e7bed-ovnkube-script-lib\") pod \"ovnkube-node-tbknb\" (UID: \"e2bb252b-36a7-4723-b3bd-e3eb873e7bed\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" Jul 13 21:59:56.639802 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.639402 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55ac7566-430e-4e45-ab69-fe943cfb4c44-metrics-certs\") pod \"network-metrics-daemon-87w2d\" (UID: \"55ac7566-430e-4e45-ab69-fe943cfb4c44\") " pod="openshift-multus/network-metrics-daemon-87w2d" Jul 13 21:59:56.639802 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.639415 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cf29609f-4a54-4c17-a7ca-10a0bdab7d28-socket-dir\") pod \"aws-ebs-csi-driver-node-9vt7w\" (UID: \"cf29609f-4a54-4c17-a7ca-10a0bdab7d28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vt7w" Jul 13 21:59:56.639802 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.639427 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cf29609f-4a54-4c17-a7ca-10a0bdab7d28-etc-selinux\") pod \"aws-ebs-csi-driver-node-9vt7w\" (UID: \"cf29609f-4a54-4c17-a7ca-10a0bdab7d28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vt7w" Jul 13 21:59:56.639802 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.639450 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e2bb252b-36a7-4723-b3bd-e3eb873e7bed-host-slash\") pod \"ovnkube-node-tbknb\" (UID: \"e2bb252b-36a7-4723-b3bd-e3eb873e7bed\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" Jul 13 21:59:56.639802 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.639476 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e2bb252b-36a7-4723-b3bd-e3eb873e7bed-node-log\") pod \"ovnkube-node-tbknb\" (UID: \"e2bb252b-36a7-4723-b3bd-e3eb873e7bed\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" Jul 13 21:59:56.639802 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.639510 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e2bb252b-36a7-4723-b3bd-e3eb873e7bed-env-overrides\") pod \"ovnkube-node-tbknb\" (UID: \"e2bb252b-36a7-4723-b3bd-e3eb873e7bed\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" Jul 13 21:59:56.639802 ip-10-0-133-234 kubenswrapper[2555]: E0713 21:59:56.639514 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 21:59:56.639802 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.639536 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cf29609f-4a54-4c17-a7ca-10a0bdab7d28-sys-fs\") pod \"aws-ebs-csi-driver-node-9vt7w\" (UID: \"cf29609f-4a54-4c17-a7ca-10a0bdab7d28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vt7w" Jul 13 21:59:56.639802 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.639559 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e2bb252b-36a7-4723-b3bd-e3eb873e7bed-host-run-netns\") pod \"ovnkube-node-tbknb\" (UID: \"e2bb252b-36a7-4723-b3bd-e3eb873e7bed\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" Jul 13 21:59:56.639802 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.639587 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e2bb252b-36a7-4723-b3bd-e3eb873e7bed-etc-openvswitch\") pod \"ovnkube-node-tbknb\" (UID: \"e2bb252b-36a7-4723-b3bd-e3eb873e7bed\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" Jul 13 21:59:56.639802 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.639624 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/56086743-e831-415a-a2c3-6982ef099751-dbus\") pod \"global-pull-secret-syncer-5bkm9\" (UID: \"56086743-e831-415a-a2c3-6982ef099751\") " pod="kube-system/global-pull-secret-syncer-5bkm9" Jul 13 21:59:56.639802 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.639647 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e2bb252b-36a7-4723-b3bd-e3eb873e7bed-host-cni-netd\") pod \"ovnkube-node-tbknb\" (UID: \"e2bb252b-36a7-4723-b3bd-e3eb873e7bed\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" Jul 13 21:59:56.639802 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.639674 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cf29609f-4a54-4c17-a7ca-10a0bdab7d28-device-dir\") pod \"aws-ebs-csi-driver-node-9vt7w\" (UID: \"cf29609f-4a54-4c17-a7ca-10a0bdab7d28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vt7w" Jul 13 21:59:56.639802 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.639703 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2bb252b-36a7-4723-b3bd-e3eb873e7bed-host-run-ovn-kubernetes\") pod \"ovnkube-node-tbknb\" (UID: \"e2bb252b-36a7-4723-b3bd-e3eb873e7bed\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" Jul 13 21:59:56.639802 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.639709 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e2bb252b-36a7-4723-b3bd-e3eb873e7bed-host-slash\") pod \"ovnkube-node-tbknb\" (UID: \"e2bb252b-36a7-4723-b3bd-e3eb873e7bed\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" Jul 13 21:59:56.640627 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.639588 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e2bb252b-36a7-4723-b3bd-e3eb873e7bed-node-log\") pod \"ovnkube-node-tbknb\" (UID: \"e2bb252b-36a7-4723-b3bd-e3eb873e7bed\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" Jul 13 21:59:56.640627 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.639731 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e2bb252b-36a7-4723-b3bd-e3eb873e7bed-var-lib-openvswitch\") pod \"ovnkube-node-tbknb\" (UID: \"e2bb252b-36a7-4723-b3bd-e3eb873e7bed\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" Jul 13 21:59:56.640627 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.639756 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csg7d\" (UniqueName: \"kubernetes.io/projected/e2bb252b-36a7-4723-b3bd-e3eb873e7bed-kube-api-access-csg7d\") pod \"ovnkube-node-tbknb\" (UID: \"e2bb252b-36a7-4723-b3bd-e3eb873e7bed\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" Jul 13 21:59:56.640627 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.639780 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e2bb252b-36a7-4723-b3bd-e3eb873e7bed-systemd-units\") pod \"ovnkube-node-tbknb\" (UID: \"e2bb252b-36a7-4723-b3bd-e3eb873e7bed\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" Jul 13 21:59:56.640627 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.639844 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2bb252b-36a7-4723-b3bd-e3eb873e7bed-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tbknb\" (UID: \"e2bb252b-36a7-4723-b3bd-e3eb873e7bed\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" Jul 13 21:59:56.640627 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.639854 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e2bb252b-36a7-4723-b3bd-e3eb873e7bed-ovnkube-script-lib\") pod \"ovnkube-node-tbknb\" (UID: \"e2bb252b-36a7-4723-b3bd-e3eb873e7bed\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" Jul 13 21:59:56.640627 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.639874 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cf29609f-4a54-4c17-a7ca-10a0bdab7d28-registration-dir\") pod \"aws-ebs-csi-driver-node-9vt7w\" (UID: \"cf29609f-4a54-4c17-a7ca-10a0bdab7d28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vt7w" Jul 13 21:59:56.640627 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.639896 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e2bb252b-36a7-4723-b3bd-e3eb873e7bed-host-run-netns\") pod \"ovnkube-node-tbknb\" (UID: \"e2bb252b-36a7-4723-b3bd-e3eb873e7bed\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" Jul 13 21:59:56.640627 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.639901 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9985a1e9-5e38-4bea-9745-988c4524fc21-hosts-file\") pod \"node-resolver-2znr7\" (UID: \"9985a1e9-5e38-4bea-9745-988c4524fc21\") " pod="openshift-dns/node-resolver-2znr7" Jul 13 21:59:56.640627 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.639926 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/061cea64-4ced-48b8-bd3a-3c6c22fc9e77-agent-certs\") pod \"konnectivity-agent-q8965\" (UID: \"061cea64-4ced-48b8-bd3a-3c6c22fc9e77\") " pod="kube-system/konnectivity-agent-q8965" Jul 13 21:59:56.640627 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.639951 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2bb252b-36a7-4723-b3bd-e3eb873e7bed-host-kubelet\") pod \"ovnkube-node-tbknb\" (UID: \"e2bb252b-36a7-4723-b3bd-e3eb873e7bed\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" Jul 13 21:59:56.640627 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.639980 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e2bb252b-36a7-4723-b3bd-e3eb873e7bed-var-lib-openvswitch\") pod \"ovnkube-node-tbknb\" (UID: \"e2bb252b-36a7-4723-b3bd-e3eb873e7bed\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" Jul 13 21:59:56.640627 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.639985 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9985a1e9-5e38-4bea-9745-988c4524fc21-tmp-dir\") pod \"node-resolver-2znr7\" (UID: \"9985a1e9-5e38-4bea-9745-988c4524fc21\") " pod="openshift-dns/node-resolver-2znr7" Jul 13 21:59:56.640627 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.640010 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/061cea64-4ced-48b8-bd3a-3c6c22fc9e77-konnectivity-ca\") pod \"konnectivity-agent-q8965\" (UID: \"061cea64-4ced-48b8-bd3a-3c6c22fc9e77\") " pod="kube-system/konnectivity-agent-q8965" Jul 13 21:59:56.640627 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.640033 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e2bb252b-36a7-4723-b3bd-e3eb873e7bed-log-socket\") pod \"ovnkube-node-tbknb\" (UID: \"e2bb252b-36a7-4723-b3bd-e3eb873e7bed\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" Jul 13 21:59:56.640627 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.640063 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e2bb252b-36a7-4723-b3bd-e3eb873e7bed-env-overrides\") pod \"ovnkube-node-tbknb\" (UID: \"e2bb252b-36a7-4723-b3bd-e3eb873e7bed\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" Jul 13 21:59:56.640627 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.640065 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cf29609f-4a54-4c17-a7ca-10a0bdab7d28-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9vt7w\" (UID: \"cf29609f-4a54-4c17-a7ca-10a0bdab7d28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vt7w" Jul 13 21:59:56.641298 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.640072 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/56086743-e831-415a-a2c3-6982ef099751-dbus\") pod \"global-pull-secret-syncer-5bkm9\" (UID: \"56086743-e831-415a-a2c3-6982ef099751\") " pod="kube-system/global-pull-secret-syncer-5bkm9" Jul 13 21:59:56.641298 ip-10-0-133-234 kubenswrapper[2555]: E0713 21:59:56.640095 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55ac7566-430e-4e45-ab69-fe943cfb4c44-metrics-certs podName:55ac7566-430e-4e45-ab69-fe943cfb4c44 nodeName:}" failed. No retries permitted until 2026-07-13 21:59:57.14008402 +0000 UTC m=+3.133838291 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55ac7566-430e-4e45-ab69-fe943cfb4c44-metrics-certs") pod "network-metrics-daemon-87w2d" (UID: "55ac7566-430e-4e45-ab69-fe943cfb4c44") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 21:59:56.641298 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.640121 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e2bb252b-36a7-4723-b3bd-e3eb873e7bed-ovn-node-metrics-cert\") pod \"ovnkube-node-tbknb\" (UID: \"e2bb252b-36a7-4723-b3bd-e3eb873e7bed\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" Jul 13 21:59:56.641298 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.640124 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cf29609f-4a54-4c17-a7ca-10a0bdab7d28-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9vt7w\" (UID: \"cf29609f-4a54-4c17-a7ca-10a0bdab7d28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vt7w" Jul 13 21:59:56.641298 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.640153 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e2bb252b-36a7-4723-b3bd-e3eb873e7bed-systemd-units\") pod \"ovnkube-node-tbknb\" (UID: \"e2bb252b-36a7-4723-b3bd-e3eb873e7bed\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" Jul 13 21:59:56.641298 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.640153 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e2bb252b-36a7-4723-b3bd-e3eb873e7bed-host-cni-netd\") pod \"ovnkube-node-tbknb\" (UID: \"e2bb252b-36a7-4723-b3bd-e3eb873e7bed\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" Jul 13 21:59:56.641298 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.640144 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e2bb252b-36a7-4723-b3bd-e3eb873e7bed-ovnkube-config\") pod \"ovnkube-node-tbknb\" (UID: \"e2bb252b-36a7-4723-b3bd-e3eb873e7bed\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" Jul 13 21:59:56.641298 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.640219 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2bb252b-36a7-4723-b3bd-e3eb873e7bed-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tbknb\" (UID: \"e2bb252b-36a7-4723-b3bd-e3eb873e7bed\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" Jul 13 21:59:56.641298 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.640243 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cf29609f-4a54-4c17-a7ca-10a0bdab7d28-device-dir\") pod \"aws-ebs-csi-driver-node-9vt7w\" (UID: \"cf29609f-4a54-4c17-a7ca-10a0bdab7d28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vt7w" Jul 13 21:59:56.641298 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.639677 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cf29609f-4a54-4c17-a7ca-10a0bdab7d28-etc-selinux\") pod \"aws-ebs-csi-driver-node-9vt7w\" (UID: \"cf29609f-4a54-4c17-a7ca-10a0bdab7d28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vt7w" Jul 13 21:59:56.641298 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.640277 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cf29609f-4a54-4c17-a7ca-10a0bdab7d28-registration-dir\") pod \"aws-ebs-csi-driver-node-9vt7w\" (UID: \"cf29609f-4a54-4c17-a7ca-10a0bdab7d28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vt7w" Jul 13 21:59:56.641298 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.640288 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2bb252b-36a7-4723-b3bd-e3eb873e7bed-host-run-ovn-kubernetes\") pod \"ovnkube-node-tbknb\" (UID: \"e2bb252b-36a7-4723-b3bd-e3eb873e7bed\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" Jul 13 21:59:56.641298 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.640379 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9985a1e9-5e38-4bea-9745-988c4524fc21-hosts-file\") pod \"node-resolver-2znr7\" (UID: \"9985a1e9-5e38-4bea-9745-988c4524fc21\") " pod="openshift-dns/node-resolver-2znr7" Jul 13 21:59:56.641298 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.640535 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e2bb252b-36a7-4723-b3bd-e3eb873e7bed-ovnkube-config\") pod \"ovnkube-node-tbknb\" (UID: \"e2bb252b-36a7-4723-b3bd-e3eb873e7bed\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" Jul 13 21:59:56.641298 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.640575 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e2bb252b-36a7-4723-b3bd-e3eb873e7bed-etc-openvswitch\") pod \"ovnkube-node-tbknb\" (UID: \"e2bb252b-36a7-4723-b3bd-e3eb873e7bed\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" Jul 13 21:59:56.641298 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.640707 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2bb252b-36a7-4723-b3bd-e3eb873e7bed-host-kubelet\") pod \"ovnkube-node-tbknb\" (UID: \"e2bb252b-36a7-4723-b3bd-e3eb873e7bed\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" Jul 13 21:59:56.641816 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.639940 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cf29609f-4a54-4c17-a7ca-10a0bdab7d28-sys-fs\") pod \"aws-ebs-csi-driver-node-9vt7w\" (UID: \"cf29609f-4a54-4c17-a7ca-10a0bdab7d28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vt7w" Jul 13 21:59:56.641816 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.640847 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9985a1e9-5e38-4bea-9745-988c4524fc21-tmp-dir\") pod \"node-resolver-2znr7\" (UID: \"9985a1e9-5e38-4bea-9745-988c4524fc21\") " pod="openshift-dns/node-resolver-2znr7" Jul 13 21:59:56.641816 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.640911 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e2bb252b-36a7-4723-b3bd-e3eb873e7bed-log-socket\") pod \"ovnkube-node-tbknb\" (UID: \"e2bb252b-36a7-4723-b3bd-e3eb873e7bed\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" Jul 13 21:59:56.641816 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.641165 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/061cea64-4ced-48b8-bd3a-3c6c22fc9e77-konnectivity-ca\") pod \"konnectivity-agent-q8965\" (UID: \"061cea64-4ced-48b8-bd3a-3c6c22fc9e77\") " pod="kube-system/konnectivity-agent-q8965" Jul 13 21:59:56.643238 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.643220 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e2bb252b-36a7-4723-b3bd-e3eb873e7bed-ovn-node-metrics-cert\") pod \"ovnkube-node-tbknb\" (UID: \"e2bb252b-36a7-4723-b3bd-e3eb873e7bed\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" Jul 13 21:59:56.643396 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.643377 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/061cea64-4ced-48b8-bd3a-3c6c22fc9e77-agent-certs\") pod \"konnectivity-agent-q8965\" (UID: \"061cea64-4ced-48b8-bd3a-3c6c22fc9e77\") " pod="kube-system/konnectivity-agent-q8965" Jul 13 21:59:56.650848 ip-10-0-133-234 kubenswrapper[2555]: E0713 21:59:56.650785 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 21:59:56.650848 ip-10-0-133-234 kubenswrapper[2555]: E0713 21:59:56.650816 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 21:59:56.650848 ip-10-0-133-234 kubenswrapper[2555]: E0713 21:59:56.650831 2555 projected.go:196] Error preparing data for projected volume kube-api-access-h8h4k for pod openshift-network-diagnostics/network-check-target-rg8tn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 21:59:56.651039 ip-10-0-133-234 kubenswrapper[2555]: E0713 21:59:56.650882 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1b9ac92-b63a-4180-9293-86c6b0c545a0-kube-api-access-h8h4k podName:d1b9ac92-b63a-4180-9293-86c6b0c545a0 nodeName:}" failed. No retries permitted until 2026-07-13 21:59:57.150867039 +0000 UTC m=+3.144621339 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-h8h4k" (UniqueName: "kubernetes.io/projected/d1b9ac92-b63a-4180-9293-86c6b0c545a0-kube-api-access-h8h4k") pod "network-check-target-rg8tn" (UID: "d1b9ac92-b63a-4180-9293-86c6b0c545a0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 21:59:56.653508 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.653488 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csg7d\" (UniqueName: \"kubernetes.io/projected/e2bb252b-36a7-4723-b3bd-e3eb873e7bed-kube-api-access-csg7d\") pod \"ovnkube-node-tbknb\" (UID: \"e2bb252b-36a7-4723-b3bd-e3eb873e7bed\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" Jul 13 21:59:56.653641 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.653622 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7kjm\" (UniqueName: \"kubernetes.io/projected/cf29609f-4a54-4c17-a7ca-10a0bdab7d28-kube-api-access-m7kjm\") pod \"aws-ebs-csi-driver-node-9vt7w\" (UID: \"cf29609f-4a54-4c17-a7ca-10a0bdab7d28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vt7w" Jul 13 21:59:56.653788 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.653767 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lzdd\" (UniqueName: \"kubernetes.io/projected/9985a1e9-5e38-4bea-9745-988c4524fc21-kube-api-access-4lzdd\") pod \"node-resolver-2znr7\" (UID: \"9985a1e9-5e38-4bea-9745-988c4524fc21\") " pod="openshift-dns/node-resolver-2znr7" Jul 13 21:59:56.653844 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.653781 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtbf5\" (UniqueName: \"kubernetes.io/projected/55ac7566-430e-4e45-ab69-fe943cfb4c44-kube-api-access-vtbf5\") pod \"network-metrics-daemon-87w2d\" (UID: \"55ac7566-430e-4e45-ab69-fe943cfb4c44\") " pod="openshift-multus/network-metrics-daemon-87w2d" Jul 13 21:59:56.729734 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.729710 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zs8wl" Jul 13 21:59:56.737573 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.737547 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6nq95" Jul 13 21:59:56.747305 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.747286 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kr77b" Jul 13 21:59:56.754983 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.754968 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-86hvh" Jul 13 21:59:56.760733 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.760715 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6xnxl" Jul 13 21:59:56.768546 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.768515 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-q8965" Jul 13 21:59:56.776336 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.776312 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" Jul 13 21:59:56.783378 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.783360 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vt7w" Jul 13 21:59:56.790484 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.790468 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2znr7" Jul 13 21:59:56.790620 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.790605 2555 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 13 21:59:56.913987 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:56.913910 2555 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 13 21:59:57.145182 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:57.145149 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56086743-e831-415a-a2c3-6982ef099751-original-pull-secret\") pod \"global-pull-secret-syncer-5bkm9\" (UID: \"56086743-e831-415a-a2c3-6982ef099751\") " pod="kube-system/global-pull-secret-syncer-5bkm9" Jul 13 21:59:57.145345 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:57.145190 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55ac7566-430e-4e45-ab69-fe943cfb4c44-metrics-certs\") pod \"network-metrics-daemon-87w2d\" (UID: \"55ac7566-430e-4e45-ab69-fe943cfb4c44\") " pod="openshift-multus/network-metrics-daemon-87w2d" Jul 13 21:59:57.145345 ip-10-0-133-234 kubenswrapper[2555]: E0713 21:59:57.145294 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 21:59:57.145345 ip-10-0-133-234 kubenswrapper[2555]: E0713 21:59:57.145294 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 21:59:57.145473 ip-10-0-133-234 kubenswrapper[2555]: E0713 21:59:57.145356 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55ac7566-430e-4e45-ab69-fe943cfb4c44-metrics-certs podName:55ac7566-430e-4e45-ab69-fe943cfb4c44 nodeName:}" failed. No retries permitted until 2026-07-13 21:59:58.145339843 +0000 UTC m=+4.139094135 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55ac7566-430e-4e45-ab69-fe943cfb4c44-metrics-certs") pod "network-metrics-daemon-87w2d" (UID: "55ac7566-430e-4e45-ab69-fe943cfb4c44") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 21:59:57.145473 ip-10-0-133-234 kubenswrapper[2555]: E0713 21:59:57.145374 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56086743-e831-415a-a2c3-6982ef099751-original-pull-secret podName:56086743-e831-415a-a2c3-6982ef099751 nodeName:}" failed. No retries permitted until 2026-07-13 21:59:58.145364928 +0000 UTC m=+4.139119213 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/56086743-e831-415a-a2c3-6982ef099751-original-pull-secret") pod "global-pull-secret-syncer-5bkm9" (UID: "56086743-e831-415a-a2c3-6982ef099751") : object "kube-system"/"original-pull-secret" not registered Jul 13 21:59:57.245677 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:57.245645 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8h4k\" (UniqueName: \"kubernetes.io/projected/d1b9ac92-b63a-4180-9293-86c6b0c545a0-kube-api-access-h8h4k\") pod \"network-check-target-rg8tn\" (UID: \"d1b9ac92-b63a-4180-9293-86c6b0c545a0\") " pod="openshift-network-diagnostics/network-check-target-rg8tn" Jul 13 21:59:57.245851 ip-10-0-133-234 kubenswrapper[2555]: E0713 21:59:57.245833 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 21:59:57.245893 ip-10-0-133-234 kubenswrapper[2555]: E0713 21:59:57.245855 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 21:59:57.245893 ip-10-0-133-234 kubenswrapper[2555]: E0713 21:59:57.245865 2555 projected.go:196] Error preparing data for projected volume kube-api-access-h8h4k for pod openshift-network-diagnostics/network-check-target-rg8tn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 21:59:57.245965 ip-10-0-133-234 kubenswrapper[2555]: E0713 21:59:57.245915 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1b9ac92-b63a-4180-9293-86c6b0c545a0-kube-api-access-h8h4k podName:d1b9ac92-b63a-4180-9293-86c6b0c545a0 nodeName:}" failed. No retries permitted until 2026-07-13 21:59:58.245899724 +0000 UTC m=+4.239653996 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-h8h4k" (UniqueName: "kubernetes.io/projected/d1b9ac92-b63a-4180-9293-86c6b0c545a0-kube-api-access-h8h4k") pod "network-check-target-rg8tn" (UID: "d1b9ac92-b63a-4180-9293-86c6b0c545a0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 21:59:57.412236 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:57.412179 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd94f40ee_a429_4bf2_b7cb_c436922cdedc.slice/crio-3eea6bd3d771faf6d4365fe2a578016c9f61b1ab3199cc2a9d60335c5f473a46 WatchSource:0}: Error finding container 3eea6bd3d771faf6d4365fe2a578016c9f61b1ab3199cc2a9d60335c5f473a46: Status 404 returned error can't find the container with id 3eea6bd3d771faf6d4365fe2a578016c9f61b1ab3199cc2a9d60335c5f473a46 Jul 13 21:59:57.413405 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:57.413379 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod888489a9_c409_4275_8318_ffee8ce2a187.slice/crio-5dacdd877f4fb474a395fe8b3279ebb7fd061733f14d17e76f0870509348f19d WatchSource:0}: Error finding container 5dacdd877f4fb474a395fe8b3279ebb7fd061733f14d17e76f0870509348f19d: Status 404 returned error can't find the container with id 5dacdd877f4fb474a395fe8b3279ebb7fd061733f14d17e76f0870509348f19d Jul 13 21:59:57.414990 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:57.414913 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22a82d33_c9c8_445a_a814_5534be9dd95e.slice/crio-88fc1dd03ee8401fc0be8623e3874fce6225ea322c4361ce90a1441b4407d241 WatchSource:0}: Error finding container 88fc1dd03ee8401fc0be8623e3874fce6225ea322c4361ce90a1441b4407d241: Status 404 returned error can't find the container with id 88fc1dd03ee8401fc0be8623e3874fce6225ea322c4361ce90a1441b4407d241 Jul 13 21:59:57.416089 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:57.416060 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9985a1e9_5e38_4bea_9745_988c4524fc21.slice/crio-807d4326bf9685fbd9ea59bc357370c6f46b6c66961ef8783add63c0cf213080 WatchSource:0}: Error finding container 807d4326bf9685fbd9ea59bc357370c6f46b6c66961ef8783add63c0cf213080: Status 404 returned error can't find the container with id 807d4326bf9685fbd9ea59bc357370c6f46b6c66961ef8783add63c0cf213080 Jul 13 21:59:57.418496 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:57.418418 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod061cea64_4ced_48b8_bd3a_3c6c22fc9e77.slice/crio-5487c6979fcabd3e0f3a6f7ff04064db083174b0c22d7c0424e03e707ab4f449 WatchSource:0}: Error finding container 5487c6979fcabd3e0f3a6f7ff04064db083174b0c22d7c0424e03e707ab4f449: Status 404 returned error can't find the container with id 5487c6979fcabd3e0f3a6f7ff04064db083174b0c22d7c0424e03e707ab4f449 Jul 13 21:59:57.430446 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:57.420775 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2bb252b_36a7_4723_b3bd_e3eb873e7bed.slice/crio-d2569bd71d2846c643d82f14031d496d9ee039d903df487d299ba901e57f40be WatchSource:0}: Error finding container d2569bd71d2846c643d82f14031d496d9ee039d903df487d299ba901e57f40be: Status 404 returned error can't find the container with id d2569bd71d2846c643d82f14031d496d9ee039d903df487d299ba901e57f40be Jul 13 21:59:57.430446 ip-10-0-133-234 kubenswrapper[2555]: W0713 21:59:57.422567 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf29609f_4a54_4c17_a7ca_10a0bdab7d28.slice/crio-4b097ecd3a371aef6f44410f34c4b94df12d52f415d4af2650981b77fe4c088f WatchSource:0}: Error finding container 4b097ecd3a371aef6f44410f34c4b94df12d52f415d4af2650981b77fe4c088f: Status 404 returned error can't find the container with id 4b097ecd3a371aef6f44410f34c4b94df12d52f415d4af2650981b77fe4c088f Jul 13 21:59:57.470917 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:57.470753 2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-12 21:54:56 +0000 UTC" deadline="2028-02-09 05:44:55.467305584 +0000 UTC" Jul 13 21:59:57.470917 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:57.470918 2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13807h44m57.996392045s" Jul 13 21:59:57.554046 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:57.554024 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5bkm9" Jul 13 21:59:57.554167 ip-10-0-133-234 kubenswrapper[2555]: E0713 21:59:57.554119 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5bkm9" podUID="56086743-e831-415a-a2c3-6982ef099751" Jul 13 21:59:57.562635 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:57.562607 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-234.ec2.internal" event={"ID":"2df761e96899302463c4ca9d4dd2ee01","Type":"ContainerStarted","Data":"61ba25c774a281e49fa52de4c95492b36016688dcadee9f4740c06c15cfa48a8"} Jul 13 21:59:57.563617 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:57.563592 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zs8wl" event={"ID":"a8651358-951f-4485-b3d8-9e21c26b7fc4","Type":"ContainerStarted","Data":"b745b90537aeb0a58f7dc0afc9653d851cdbe761969db82a8d218110dfe220bc"} Jul 13 21:59:57.564704 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:57.564681 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6nq95" event={"ID":"54550ef2-07f8-4e03-b898-c5d7536cc61e","Type":"ContainerStarted","Data":"4557ae805e5a7246f78e3e4d683132c5825cf8e6f1c71a051a29712f182e292d"} Jul 13 21:59:57.567080 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:57.567060 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" event={"ID":"e2bb252b-36a7-4723-b3bd-e3eb873e7bed","Type":"ContainerStarted","Data":"d2569bd71d2846c643d82f14031d496d9ee039d903df487d299ba901e57f40be"} Jul 13 21:59:57.568084 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:57.568060 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-q8965" event={"ID":"061cea64-4ced-48b8-bd3a-3c6c22fc9e77","Type":"ContainerStarted","Data":"5487c6979fcabd3e0f3a6f7ff04064db083174b0c22d7c0424e03e707ab4f449"} Jul 13 21:59:57.569105 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:57.569085 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-86hvh" event={"ID":"22a82d33-c9c8-445a-a814-5534be9dd95e","Type":"ContainerStarted","Data":"88fc1dd03ee8401fc0be8623e3874fce6225ea322c4361ce90a1441b4407d241"} Jul 13 21:59:57.570118 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:57.570099 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vt7w" event={"ID":"cf29609f-4a54-4c17-a7ca-10a0bdab7d28","Type":"ContainerStarted","Data":"4b097ecd3a371aef6f44410f34c4b94df12d52f415d4af2650981b77fe4c088f"} Jul 13 21:59:57.572020 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:57.572001 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2znr7" event={"ID":"9985a1e9-5e38-4bea-9745-988c4524fc21","Type":"ContainerStarted","Data":"807d4326bf9685fbd9ea59bc357370c6f46b6c66961ef8783add63c0cf213080"} Jul 13 21:59:57.573034 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:57.573016 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kr77b" event={"ID":"888489a9-c409-4275-8318-ffee8ce2a187","Type":"ContainerStarted","Data":"5dacdd877f4fb474a395fe8b3279ebb7fd061733f14d17e76f0870509348f19d"} Jul 13 21:59:57.575438 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:57.575415 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6xnxl" event={"ID":"d94f40ee-a429-4bf2-b7cb-c436922cdedc","Type":"ContainerStarted","Data":"3eea6bd3d771faf6d4365fe2a578016c9f61b1ab3199cc2a9d60335c5f473a46"} Jul 13 21:59:57.576556 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:57.576517 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-234.ec2.internal" podStartSLOduration=1.57650495 podStartE2EDuration="1.57650495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 21:59:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:59:57.576247977 +0000 UTC m=+3.570002291" watchObservedRunningTime="2026-07-13 21:59:57.57650495 +0000 UTC m=+3.570259244" Jul 13 21:59:58.154326 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:58.151819 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55ac7566-430e-4e45-ab69-fe943cfb4c44-metrics-certs\") pod \"network-metrics-daemon-87w2d\" (UID: \"55ac7566-430e-4e45-ab69-fe943cfb4c44\") " pod="openshift-multus/network-metrics-daemon-87w2d" Jul 13 21:59:58.154326 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:58.151909 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56086743-e831-415a-a2c3-6982ef099751-original-pull-secret\") pod \"global-pull-secret-syncer-5bkm9\" (UID: \"56086743-e831-415a-a2c3-6982ef099751\") " pod="kube-system/global-pull-secret-syncer-5bkm9" Jul 13 21:59:58.154326 ip-10-0-133-234 kubenswrapper[2555]: E0713 21:59:58.152051 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 21:59:58.154326 ip-10-0-133-234 kubenswrapper[2555]: E0713 21:59:58.152124 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56086743-e831-415a-a2c3-6982ef099751-original-pull-secret podName:56086743-e831-415a-a2c3-6982ef099751 nodeName:}" failed. No retries permitted until 2026-07-13 22:00:00.152104458 +0000 UTC m=+6.145858731 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/56086743-e831-415a-a2c3-6982ef099751-original-pull-secret") pod "global-pull-secret-syncer-5bkm9" (UID: "56086743-e831-415a-a2c3-6982ef099751") : object "kube-system"/"original-pull-secret" not registered Jul 13 21:59:58.154326 ip-10-0-133-234 kubenswrapper[2555]: E0713 21:59:58.152192 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 21:59:58.154326 ip-10-0-133-234 kubenswrapper[2555]: E0713 21:59:58.152243 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55ac7566-430e-4e45-ab69-fe943cfb4c44-metrics-certs podName:55ac7566-430e-4e45-ab69-fe943cfb4c44 nodeName:}" failed. No retries permitted until 2026-07-13 22:00:00.152232228 +0000 UTC m=+6.145986504 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55ac7566-430e-4e45-ab69-fe943cfb4c44-metrics-certs") pod "network-metrics-daemon-87w2d" (UID: "55ac7566-430e-4e45-ab69-fe943cfb4c44") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 21:59:58.252593 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:58.252556 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8h4k\" (UniqueName: \"kubernetes.io/projected/d1b9ac92-b63a-4180-9293-86c6b0c545a0-kube-api-access-h8h4k\") pod \"network-check-target-rg8tn\" (UID: \"d1b9ac92-b63a-4180-9293-86c6b0c545a0\") " pod="openshift-network-diagnostics/network-check-target-rg8tn" Jul 13 21:59:58.252983 ip-10-0-133-234 kubenswrapper[2555]: E0713 21:59:58.252773 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 21:59:58.252983 ip-10-0-133-234 kubenswrapper[2555]: E0713 21:59:58.252796 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 21:59:58.252983 ip-10-0-133-234 kubenswrapper[2555]: E0713 21:59:58.252810 2555 projected.go:196] Error preparing data for projected volume kube-api-access-h8h4k for pod openshift-network-diagnostics/network-check-target-rg8tn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 21:59:58.252983 ip-10-0-133-234 kubenswrapper[2555]: E0713 21:59:58.252868 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1b9ac92-b63a-4180-9293-86c6b0c545a0-kube-api-access-h8h4k podName:d1b9ac92-b63a-4180-9293-86c6b0c545a0 nodeName:}" failed. No retries permitted until 2026-07-13 22:00:00.252849889 +0000 UTC m=+6.246604175 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-h8h4k" (UniqueName: "kubernetes.io/projected/d1b9ac92-b63a-4180-9293-86c6b0c545a0-kube-api-access-h8h4k") pod "network-check-target-rg8tn" (UID: "d1b9ac92-b63a-4180-9293-86c6b0c545a0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 21:59:58.471531 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:58.471441 2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-12 21:54:56 +0000 UTC" deadline="2028-01-22 19:42:16.681216461 +0000 UTC" Jul 13 21:59:58.471531 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:58.471484 2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13389h42m18.209737777s" Jul 13 21:59:58.557348 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:58.557317 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rg8tn" Jul 13 21:59:58.557511 ip-10-0-133-234 kubenswrapper[2555]: E0713 21:59:58.557452 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rg8tn" podUID="d1b9ac92-b63a-4180-9293-86c6b0c545a0" Jul 13 21:59:58.557580 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:58.557508 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87w2d" Jul 13 21:59:58.557673 ip-10-0-133-234 kubenswrapper[2555]: E0713 21:59:58.557631 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87w2d" podUID="55ac7566-430e-4e45-ab69-fe943cfb4c44" Jul 13 21:59:58.596844 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:58.595719 2555 generic.go:356] "Generic (PLEG): container finished" podID="6fba7a483df990721f423a08a6cb4680" containerID="c0b260e27fe4d530266790995b2bfa33fd2b610207e49d50770d9e73bdb93fa7" exitCode=0 Jul 13 21:59:58.596844 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:58.596792 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-234.ec2.internal" event={"ID":"6fba7a483df990721f423a08a6cb4680","Type":"ContainerDied","Data":"c0b260e27fe4d530266790995b2bfa33fd2b610207e49d50770d9e73bdb93fa7"} Jul 13 21:59:59.553968 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:59.553386 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5bkm9" Jul 13 21:59:59.553968 ip-10-0-133-234 kubenswrapper[2555]: E0713 21:59:59.553539 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5bkm9" podUID="56086743-e831-415a-a2c3-6982ef099751" Jul 13 21:59:59.607779 ip-10-0-133-234 kubenswrapper[2555]: I0713 21:59:59.607741 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-234.ec2.internal" event={"ID":"6fba7a483df990721f423a08a6cb4680","Type":"ContainerStarted","Data":"3cde6ec278b23e893a390f5d8e1fcba0eb8938ef9b5265e472040a4cd45cd6e2"} Jul 13 22:00:00.167802 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:00.167762 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56086743-e831-415a-a2c3-6982ef099751-original-pull-secret\") pod \"global-pull-secret-syncer-5bkm9\" (UID: \"56086743-e831-415a-a2c3-6982ef099751\") " pod="kube-system/global-pull-secret-syncer-5bkm9" Jul 13 22:00:00.167997 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:00.167831 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55ac7566-430e-4e45-ab69-fe943cfb4c44-metrics-certs\") pod \"network-metrics-daemon-87w2d\" (UID: \"55ac7566-430e-4e45-ab69-fe943cfb4c44\") " pod="openshift-multus/network-metrics-daemon-87w2d" Jul 13 22:00:00.168072 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:00.168007 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 22:00:00.168072 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:00.168070 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55ac7566-430e-4e45-ab69-fe943cfb4c44-metrics-certs podName:55ac7566-430e-4e45-ab69-fe943cfb4c44 nodeName:}" failed. No retries permitted until 2026-07-13 22:00:04.168051618 +0000 UTC m=+10.161805894 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55ac7566-430e-4e45-ab69-fe943cfb4c44-metrics-certs") pod "network-metrics-daemon-87w2d" (UID: "55ac7566-430e-4e45-ab69-fe943cfb4c44") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 22:00:00.168504 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:00.168483 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 22:00:00.168613 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:00.168547 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56086743-e831-415a-a2c3-6982ef099751-original-pull-secret podName:56086743-e831-415a-a2c3-6982ef099751 nodeName:}" failed. No retries permitted until 2026-07-13 22:00:04.168531802 +0000 UTC m=+10.162286091 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/56086743-e831-415a-a2c3-6982ef099751-original-pull-secret") pod "global-pull-secret-syncer-5bkm9" (UID: "56086743-e831-415a-a2c3-6982ef099751") : object "kube-system"/"original-pull-secret" not registered Jul 13 22:00:00.269232 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:00.269172 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8h4k\" (UniqueName: \"kubernetes.io/projected/d1b9ac92-b63a-4180-9293-86c6b0c545a0-kube-api-access-h8h4k\") pod \"network-check-target-rg8tn\" (UID: \"d1b9ac92-b63a-4180-9293-86c6b0c545a0\") " pod="openshift-network-diagnostics/network-check-target-rg8tn" Jul 13 22:00:00.269415 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:00.269359 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 22:00:00.269415 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:00.269385 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 22:00:00.269415 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:00.269399 2555 projected.go:196] Error preparing data for projected volume kube-api-access-h8h4k for pod openshift-network-diagnostics/network-check-target-rg8tn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 22:00:00.269582 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:00.269458 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1b9ac92-b63a-4180-9293-86c6b0c545a0-kube-api-access-h8h4k podName:d1b9ac92-b63a-4180-9293-86c6b0c545a0 nodeName:}" failed. No retries permitted until 2026-07-13 22:00:04.269437524 +0000 UTC m=+10.263191812 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-h8h4k" (UniqueName: "kubernetes.io/projected/d1b9ac92-b63a-4180-9293-86c6b0c545a0-kube-api-access-h8h4k") pod "network-check-target-rg8tn" (UID: "d1b9ac92-b63a-4180-9293-86c6b0c545a0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 22:00:00.554407 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:00.554324 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rg8tn" Jul 13 22:00:00.554841 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:00.554454 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rg8tn" podUID="d1b9ac92-b63a-4180-9293-86c6b0c545a0" Jul 13 22:00:00.555091 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:00.554957 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87w2d" Jul 13 22:00:00.555091 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:00.555070 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87w2d" podUID="55ac7566-430e-4e45-ab69-fe943cfb4c44" Jul 13 22:00:01.554236 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:01.553549 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5bkm9" Jul 13 22:00:01.554236 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:01.553651 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5bkm9" podUID="56086743-e831-415a-a2c3-6982ef099751" Jul 13 22:00:02.554096 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:02.553533 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rg8tn" Jul 13 22:00:02.554096 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:02.553669 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rg8tn" podUID="d1b9ac92-b63a-4180-9293-86c6b0c545a0" Jul 13 22:00:02.554339 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:02.554138 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87w2d" Jul 13 22:00:02.554339 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:02.554281 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87w2d" podUID="55ac7566-430e-4e45-ab69-fe943cfb4c44" Jul 13 22:00:03.554180 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:03.554143 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5bkm9" Jul 13 22:00:03.554609 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:03.554308 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5bkm9" podUID="56086743-e831-415a-a2c3-6982ef099751" Jul 13 22:00:04.205817 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:04.205781 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56086743-e831-415a-a2c3-6982ef099751-original-pull-secret\") pod \"global-pull-secret-syncer-5bkm9\" (UID: \"56086743-e831-415a-a2c3-6982ef099751\") " pod="kube-system/global-pull-secret-syncer-5bkm9" Jul 13 22:00:04.205978 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:04.205855 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55ac7566-430e-4e45-ab69-fe943cfb4c44-metrics-certs\") pod \"network-metrics-daemon-87w2d\" (UID: \"55ac7566-430e-4e45-ab69-fe943cfb4c44\") " pod="openshift-multus/network-metrics-daemon-87w2d" Jul 13 22:00:04.205978 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:04.205953 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 22:00:04.205978 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:04.205962 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 22:00:04.206129 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:04.206009 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55ac7566-430e-4e45-ab69-fe943cfb4c44-metrics-certs podName:55ac7566-430e-4e45-ab69-fe943cfb4c44 nodeName:}" failed. No retries permitted until 2026-07-13 22:00:12.205996522 +0000 UTC m=+18.199750806 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55ac7566-430e-4e45-ab69-fe943cfb4c44-metrics-certs") pod "network-metrics-daemon-87w2d" (UID: "55ac7566-430e-4e45-ab69-fe943cfb4c44") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 22:00:04.206129 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:04.206021 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56086743-e831-415a-a2c3-6982ef099751-original-pull-secret podName:56086743-e831-415a-a2c3-6982ef099751 nodeName:}" failed. No retries permitted until 2026-07-13 22:00:12.206015295 +0000 UTC m=+18.199769565 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/56086743-e831-415a-a2c3-6982ef099751-original-pull-secret") pod "global-pull-secret-syncer-5bkm9" (UID: "56086743-e831-415a-a2c3-6982ef099751") : object "kube-system"/"original-pull-secret" not registered Jul 13 22:00:04.306792 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:04.306753 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8h4k\" (UniqueName: \"kubernetes.io/projected/d1b9ac92-b63a-4180-9293-86c6b0c545a0-kube-api-access-h8h4k\") pod \"network-check-target-rg8tn\" (UID: \"d1b9ac92-b63a-4180-9293-86c6b0c545a0\") " pod="openshift-network-diagnostics/network-check-target-rg8tn" Jul 13 22:00:04.306948 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:04.306916 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 22:00:04.306948 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:04.306940 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 22:00:04.307034 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:04.306955 2555 projected.go:196] Error preparing data for projected volume kube-api-access-h8h4k for pod openshift-network-diagnostics/network-check-target-rg8tn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 22:00:04.307034 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:04.307011 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1b9ac92-b63a-4180-9293-86c6b0c545a0-kube-api-access-h8h4k podName:d1b9ac92-b63a-4180-9293-86c6b0c545a0 nodeName:}" failed. No retries permitted until 2026-07-13 22:00:12.306993539 +0000 UTC m=+18.300747833 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-h8h4k" (UniqueName: "kubernetes.io/projected/d1b9ac92-b63a-4180-9293-86c6b0c545a0-kube-api-access-h8h4k") pod "network-check-target-rg8tn" (UID: "d1b9ac92-b63a-4180-9293-86c6b0c545a0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 22:00:04.555292 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:04.555192 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rg8tn" Jul 13 22:00:04.555707 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:04.555326 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rg8tn" podUID="d1b9ac92-b63a-4180-9293-86c6b0c545a0" Jul 13 22:00:04.555707 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:04.555381 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87w2d" Jul 13 22:00:04.555707 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:04.555504 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87w2d" podUID="55ac7566-430e-4e45-ab69-fe943cfb4c44" Jul 13 22:00:05.553800 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:05.553764 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5bkm9" Jul 13 22:00:05.553976 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:05.553910 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5bkm9" podUID="56086743-e831-415a-a2c3-6982ef099751" Jul 13 22:00:06.553340 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:06.553304 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87w2d" Jul 13 22:00:06.553825 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:06.553311 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rg8tn" Jul 13 22:00:06.553825 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:06.553443 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87w2d" podUID="55ac7566-430e-4e45-ab69-fe943cfb4c44" Jul 13 22:00:06.553825 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:06.553483 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rg8tn" podUID="d1b9ac92-b63a-4180-9293-86c6b0c545a0" Jul 13 22:00:07.554170 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:07.554126 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5bkm9" Jul 13 22:00:07.554570 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:07.554315 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5bkm9" podUID="56086743-e831-415a-a2c3-6982ef099751" Jul 13 22:00:08.553741 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:08.553670 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87w2d" Jul 13 22:00:08.553888 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:08.553667 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rg8tn" Jul 13 22:00:08.553888 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:08.553799 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87w2d" podUID="55ac7566-430e-4e45-ab69-fe943cfb4c44" Jul 13 22:00:08.553888 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:08.553871 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rg8tn" podUID="d1b9ac92-b63a-4180-9293-86c6b0c545a0" Jul 13 22:00:09.554033 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:09.553994 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5bkm9" Jul 13 22:00:09.554479 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:09.554148 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5bkm9" podUID="56086743-e831-415a-a2c3-6982ef099751" Jul 13 22:00:10.553823 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:10.553447 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rg8tn" Jul 13 22:00:10.553823 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:10.553463 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87w2d" Jul 13 22:00:10.553823 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:10.553585 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rg8tn" podUID="d1b9ac92-b63a-4180-9293-86c6b0c545a0" Jul 13 22:00:10.553823 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:10.553720 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87w2d" podUID="55ac7566-430e-4e45-ab69-fe943cfb4c44" Jul 13 22:00:11.553769 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:11.553733 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5bkm9" Jul 13 22:00:11.553990 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:11.553847 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5bkm9" podUID="56086743-e831-415a-a2c3-6982ef099751" Jul 13 22:00:12.268077 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:12.268037 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56086743-e831-415a-a2c3-6982ef099751-original-pull-secret\") pod \"global-pull-secret-syncer-5bkm9\" (UID: \"56086743-e831-415a-a2c3-6982ef099751\") " pod="kube-system/global-pull-secret-syncer-5bkm9" Jul 13 22:00:12.268517 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:12.268102 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55ac7566-430e-4e45-ab69-fe943cfb4c44-metrics-certs\") pod \"network-metrics-daemon-87w2d\" (UID: \"55ac7566-430e-4e45-ab69-fe943cfb4c44\") " pod="openshift-multus/network-metrics-daemon-87w2d" Jul 13 22:00:12.268517 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:12.268233 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 22:00:12.268517 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:12.268251 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 22:00:12.268517 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:12.268310 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56086743-e831-415a-a2c3-6982ef099751-original-pull-secret podName:56086743-e831-415a-a2c3-6982ef099751 nodeName:}" failed. No retries permitted until 2026-07-13 22:00:28.268288971 +0000 UTC m=+34.262043257 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/56086743-e831-415a-a2c3-6982ef099751-original-pull-secret") pod "global-pull-secret-syncer-5bkm9" (UID: "56086743-e831-415a-a2c3-6982ef099751") : object "kube-system"/"original-pull-secret" not registered Jul 13 22:00:12.268517 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:12.268330 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55ac7566-430e-4e45-ab69-fe943cfb4c44-metrics-certs podName:55ac7566-430e-4e45-ab69-fe943cfb4c44 nodeName:}" failed. No retries permitted until 2026-07-13 22:00:28.268319352 +0000 UTC m=+34.262073628 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55ac7566-430e-4e45-ab69-fe943cfb4c44-metrics-certs") pod "network-metrics-daemon-87w2d" (UID: "55ac7566-430e-4e45-ab69-fe943cfb4c44") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 22:00:12.369379 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:12.369340 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8h4k\" (UniqueName: \"kubernetes.io/projected/d1b9ac92-b63a-4180-9293-86c6b0c545a0-kube-api-access-h8h4k\") pod \"network-check-target-rg8tn\" (UID: \"d1b9ac92-b63a-4180-9293-86c6b0c545a0\") " pod="openshift-network-diagnostics/network-check-target-rg8tn" Jul 13 22:00:12.369552 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:12.369519 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 22:00:12.369552 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:12.369547 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 22:00:12.369637 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:12.369562 2555 projected.go:196] Error preparing data for projected volume kube-api-access-h8h4k for pod openshift-network-diagnostics/network-check-target-rg8tn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 22:00:12.369637 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:12.369629 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1b9ac92-b63a-4180-9293-86c6b0c545a0-kube-api-access-h8h4k podName:d1b9ac92-b63a-4180-9293-86c6b0c545a0 nodeName:}" failed. No retries permitted until 2026-07-13 22:00:28.369609854 +0000 UTC m=+34.363364173 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-h8h4k" (UniqueName: "kubernetes.io/projected/d1b9ac92-b63a-4180-9293-86c6b0c545a0-kube-api-access-h8h4k") pod "network-check-target-rg8tn" (UID: "d1b9ac92-b63a-4180-9293-86c6b0c545a0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 22:00:12.555819 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:12.555310 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rg8tn" Jul 13 22:00:12.555819 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:12.555445 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rg8tn" podUID="d1b9ac92-b63a-4180-9293-86c6b0c545a0" Jul 13 22:00:12.555819 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:12.555535 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87w2d" Jul 13 22:00:12.555819 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:12.555650 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87w2d" podUID="55ac7566-430e-4e45-ab69-fe943cfb4c44" Jul 13 22:00:13.554123 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:13.554080 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5bkm9" Jul 13 22:00:13.554579 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:13.554271 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5bkm9" podUID="56086743-e831-415a-a2c3-6982ef099751" Jul 13 22:00:14.555027 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:14.554983 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rg8tn" Jul 13 22:00:14.555430 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:14.555112 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rg8tn" podUID="d1b9ac92-b63a-4180-9293-86c6b0c545a0" Jul 13 22:00:14.555489 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:14.555471 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87w2d" Jul 13 22:00:14.555577 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:14.555550 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87w2d" podUID="55ac7566-430e-4e45-ab69-fe943cfb4c44" Jul 13 22:00:15.553579 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:15.553545 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5bkm9" Jul 13 22:00:15.553745 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:15.553688 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5bkm9" podUID="56086743-e831-415a-a2c3-6982ef099751" Jul 13 22:00:16.554664 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:16.554334 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rg8tn" Jul 13 22:00:16.555434 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:16.554669 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rg8tn" podUID="d1b9ac92-b63a-4180-9293-86c6b0c545a0" Jul 13 22:00:16.555434 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:16.554808 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87w2d" Jul 13 22:00:16.555434 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:16.554897 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87w2d" podUID="55ac7566-430e-4e45-ab69-fe943cfb4c44" Jul 13 22:00:16.644065 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:16.643707 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6nq95" event={"ID":"54550ef2-07f8-4e03-b898-c5d7536cc61e","Type":"ContainerStarted","Data":"186d9a91d0b234db2901f2693ba52b5c70bb9fb476358f2044452964e14553a6"} Jul 13 22:00:16.646105 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:16.646080 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" event={"ID":"e2bb252b-36a7-4723-b3bd-e3eb873e7bed","Type":"ContainerStarted","Data":"aaa8261dc5aaf106b54fa3ffd4f067a86d3667f39b3e8187c428a5d4e5e7a9bd"} Jul 13 22:00:16.646229 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:16.646168 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" event={"ID":"e2bb252b-36a7-4723-b3bd-e3eb873e7bed","Type":"ContainerStarted","Data":"0371190d34edce0653fec87338188b7383e636794dff4208a8f342bb1b861a77"} Jul 13 22:00:16.647724 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:16.647697 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-q8965" event={"ID":"061cea64-4ced-48b8-bd3a-3c6c22fc9e77","Type":"ContainerStarted","Data":"a75dba4f448ea27bdf1e3de797205fe6b88f9b4078623b27bc589df3bf9c2030"} Jul 13 22:00:16.649143 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:16.649116 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-86hvh" event={"ID":"22a82d33-c9c8-445a-a814-5534be9dd95e","Type":"ContainerStarted","Data":"af342ec2ab627578d95cff781dcf98bd7ab5c601b40ad8bb7688830c688c61b3"} Jul 13 22:00:16.650505 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:16.650484 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vt7w" event={"ID":"cf29609f-4a54-4c17-a7ca-10a0bdab7d28","Type":"ContainerStarted","Data":"09bc8620011a2e4dd7112b28392d6e09abdeef3b48f60b329be0a402e9422f4f"} Jul 13 22:00:16.651895 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:16.651835 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2znr7" event={"ID":"9985a1e9-5e38-4bea-9745-988c4524fc21","Type":"ContainerStarted","Data":"0ee6388b010da8708a75e12fa8dae7c6b9bc686bac1526a10606dac429d2757a"} Jul 13 22:00:16.653638 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:16.653613 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kr77b" event={"ID":"888489a9-c409-4275-8318-ffee8ce2a187","Type":"ContainerStarted","Data":"d6b18948c21d641e3cac57f46d1d350465c519037bd615c71c94edfc9984c18b"} Jul 13 22:00:16.655377 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:16.655353 2555 generic.go:356] "Generic (PLEG): container finished" podID="d94f40ee-a429-4bf2-b7cb-c436922cdedc" containerID="01e0e06de4517797b0220cbb7ef2d7f5b37f76eeb139a2fe0da82a219fcef825" exitCode=0 Jul 13 22:00:16.655474 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:16.655391 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6xnxl" event={"ID":"d94f40ee-a429-4bf2-b7cb-c436922cdedc","Type":"ContainerDied","Data":"01e0e06de4517797b0220cbb7ef2d7f5b37f76eeb139a2fe0da82a219fcef825"} Jul 13 22:00:16.658513 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:16.658463 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-6nq95" podStartSLOduration=3.746367676 podStartE2EDuration="22.658450359s" totalImagesPullingTime="18.912082683s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 21:59:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:00:16.657680488 +0000 UTC m=+22.651434791" watchObservedRunningTime="2026-07-13 22:00:16.658450359 +0000 UTC m=+22.652204652" Jul 13 22:00:16.658604 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:16.658534 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-234.ec2.internal" podStartSLOduration=20.658529068 podStartE2EDuration="20.658529068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 21:59:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:59:59.625315228 +0000 UTC m=+5.619069530" watchObservedRunningTime="2026-07-13 22:00:16.658529068 +0000 UTC m=+22.652283382" Jul 13 22:00:16.696497 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:16.696455 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-q8965" podStartSLOduration=11.902399484 podStartE2EDuration="22.696438847s" totalImagesPullingTime="10.794039363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:59:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:00:16.69631713 +0000 UTC m=+22.690071444" watchObservedRunningTime="2026-07-13 22:00:16.696438847 +0000 UTC m=+22.690193140" Jul 13 22:00:16.729723 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:16.729610 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-2znr7" podStartSLOduration=3.80570682 podStartE2EDuration="22.729590735s" totalImagesPullingTime="18.923883915s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 21:59:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:00:16.729297413 +0000 UTC m=+22.723051727" watchObservedRunningTime="2026-07-13 22:00:16.729590735 +0000 UTC m=+22.723345029" Jul 13 22:00:16.729905 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:16.729872 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-kr77b" podStartSLOduration=3.8014638080000003 podStartE2EDuration="22.729864161s" totalImagesPullingTime="18.928400353s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 21:59:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:00:16.715267353 +0000 UTC m=+22.709021665" watchObservedRunningTime="2026-07-13 22:00:16.729864161 +0000 UTC m=+22.723618455" Jul 13 22:00:16.745843 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:16.745805 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-86hvh" podStartSLOduration=3.759676122 podStartE2EDuration="22.745791055s" totalImagesPullingTime="18.986114933s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 21:59:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:00:16.745344816 +0000 UTC m=+22.739099132" watchObservedRunningTime="2026-07-13 22:00:16.745791055 +0000 UTC m=+22.739545348" Jul 13 22:00:17.553732 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:17.553702 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5bkm9" Jul 13 22:00:17.553910 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:17.553807 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5bkm9" podUID="56086743-e831-415a-a2c3-6982ef099751" Jul 13 22:00:17.662511 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:17.662480 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" event={"ID":"e2bb252b-36a7-4723-b3bd-e3eb873e7bed","Type":"ContainerStarted","Data":"5b258d92554c569c47a062a0d11e80e9bd24a16070325a3cfb07f681b7af199d"} Jul 13 22:00:17.663016 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:17.662559 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" event={"ID":"e2bb252b-36a7-4723-b3bd-e3eb873e7bed","Type":"ContainerStarted","Data":"ce705809464166c8fa9c417eee3a117878419d5f012819229a922a999398ffc8"} Jul 13 22:00:17.663016 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:17.662599 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" event={"ID":"e2bb252b-36a7-4723-b3bd-e3eb873e7bed","Type":"ContainerStarted","Data":"81b46b7f1f7f0c375f22cf13b5b6e8d70f780ad874e748796fce14d7f4d5aa6f"} Jul 13 22:00:17.663016 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:17.662629 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" event={"ID":"e2bb252b-36a7-4723-b3bd-e3eb873e7bed","Type":"ContainerStarted","Data":"7ed21926107899594bd48a8ef7996fc0cfcedc7b3d099928b4f1e1dc219b2f6d"} Jul 13 22:00:17.827036 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:17.827006 2555 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 13 22:00:18.491476 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:18.491360 2555 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-13T22:00:17.827031134Z","UUID":"3a84e4eb-4d51-41e1-8f61-a8de739b40a2","Handler":null,"Name":"","Endpoint":""} Jul 13 22:00:18.493062 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:18.493037 2555 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 13 22:00:18.493191 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:18.493084 2555 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 13 22:00:18.553137 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:18.553108 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rg8tn" Jul 13 22:00:18.553312 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:18.553233 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rg8tn" podUID="d1b9ac92-b63a-4180-9293-86c6b0c545a0" Jul 13 22:00:18.553483 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:18.553465 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87w2d" Jul 13 22:00:18.553598 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:18.553573 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87w2d" podUID="55ac7566-430e-4e45-ab69-fe943cfb4c44" Jul 13 22:00:18.666585 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:18.666543 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zs8wl" event={"ID":"a8651358-951f-4485-b3d8-9e21c26b7fc4","Type":"ContainerStarted","Data":"f4c81252937faafc42fe83930168cfa8022823b23b927f0c3f9626a0e98e91b9"} Jul 13 22:00:18.668791 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:18.668763 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vt7w" event={"ID":"cf29609f-4a54-4c17-a7ca-10a0bdab7d28","Type":"ContainerStarted","Data":"3930ad671c6fedef02eb81a88314c5c71af43e6a67023160972fbaa6e1984e81"} Jul 13 22:00:18.685609 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:18.685563 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zs8wl" podStartSLOduration=5.773910361 podStartE2EDuration="24.685547707s" totalImagesPullingTime="18.911637346s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 21:59:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:00:18.685395449 +0000 UTC m=+24.679149763" watchObservedRunningTime="2026-07-13 22:00:18.685547707 +0000 UTC m=+24.679302002" Jul 13 22:00:19.041148 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:19.041114 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-q8965" Jul 13 22:00:19.041819 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:19.041802 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-q8965" Jul 13 22:00:19.554086 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:19.554057 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5bkm9" Jul 13 22:00:19.554368 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:19.554215 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5bkm9" podUID="56086743-e831-415a-a2c3-6982ef099751" Jul 13 22:00:19.675096 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:19.675064 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" event={"ID":"e2bb252b-36a7-4723-b3bd-e3eb873e7bed","Type":"ContainerStarted","Data":"f37f5240bdc39ac9dd189fafd5e3c9bc44f2c7f8ac485ccad9c5842fc0966366"} Jul 13 22:00:19.677445 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:19.677404 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vt7w" event={"ID":"cf29609f-4a54-4c17-a7ca-10a0bdab7d28","Type":"ContainerStarted","Data":"6652625982bf10baa74aa950e8c33848d6879538e4a51e9e07cca8332d4bc2d6"} Jul 13 22:00:19.677933 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:19.677902 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-q8965" Jul 13 22:00:19.678284 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:19.678259 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-q8965" Jul 13 22:00:19.695344 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:19.695294 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9vt7w" podStartSLOduration=4.443827063 podStartE2EDuration="25.695277521s" totalImagesPullingTime="21.251450458s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 21:59:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:00:19.694504639 +0000 UTC m=+25.688258958" watchObservedRunningTime="2026-07-13 22:00:19.695277521 +0000 UTC m=+25.689031816" Jul 13 22:00:20.554311 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:20.554059 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87w2d" Jul 13 22:00:20.554557 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:20.554110 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rg8tn" Jul 13 22:00:20.554557 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:20.554436 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87w2d" podUID="55ac7566-430e-4e45-ab69-fe943cfb4c44" Jul 13 22:00:20.554557 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:20.554468 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rg8tn" podUID="d1b9ac92-b63a-4180-9293-86c6b0c545a0" Jul 13 22:00:21.553379 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:21.553337 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5bkm9" Jul 13 22:00:21.554107 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:21.553453 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5bkm9" podUID="56086743-e831-415a-a2c3-6982ef099751" Jul 13 22:00:21.686003 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:21.685774 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" event={"ID":"e2bb252b-36a7-4723-b3bd-e3eb873e7bed","Type":"ContainerStarted","Data":"5c3b82b02aa24e9515b44115f8e8f4908bc3809824aef258fefdbb0abee96485"} Jul 13 22:00:21.686187 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:21.686164 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" Jul 13 22:00:21.686271 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:21.686263 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" Jul 13 22:00:21.686325 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:21.686314 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" Jul 13 22:00:21.687603 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:21.687579 2555 generic.go:356] "Generic (PLEG): container finished" podID="d94f40ee-a429-4bf2-b7cb-c436922cdedc" containerID="fd794c73d70ba6df7cb1de8285f510c2f9049ee2b27e2aa328b743d02cfb2f20" exitCode=0 Jul 13 22:00:21.687706 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:21.687667 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6xnxl" event={"ID":"d94f40ee-a429-4bf2-b7cb-c436922cdedc","Type":"ContainerDied","Data":"fd794c73d70ba6df7cb1de8285f510c2f9049ee2b27e2aa328b743d02cfb2f20"} Jul 13 22:00:21.702300 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:21.702267 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" Jul 13 22:00:21.702408 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:21.702364 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" Jul 13 22:00:21.711915 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:21.711881 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" podStartSLOduration=8.715618726 podStartE2EDuration="27.711869902s" totalImagesPullingTime="18.996251176s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 21:59:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:00:21.711565694 +0000 UTC m=+27.705320005" watchObservedRunningTime="2026-07-13 22:00:21.711869902 +0000 UTC m=+27.705624195" Jul 13 22:00:22.553302 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:22.553191 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rg8tn" Jul 13 22:00:22.553569 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:22.553230 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87w2d" Jul 13 22:00:22.553569 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:22.553338 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rg8tn" podUID="d1b9ac92-b63a-4180-9293-86c6b0c545a0" Jul 13 22:00:22.553569 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:22.553433 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87w2d" podUID="55ac7566-430e-4e45-ab69-fe943cfb4c44" Jul 13 22:00:23.553134 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:23.553105 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5bkm9" Jul 13 22:00:23.553322 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:23.553210 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5bkm9" podUID="56086743-e831-415a-a2c3-6982ef099751" Jul 13 22:00:23.614374 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:23.614343 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rg8tn"] Jul 13 22:00:23.614757 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:23.614469 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rg8tn" Jul 13 22:00:23.614757 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:23.614560 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rg8tn" podUID="d1b9ac92-b63a-4180-9293-86c6b0c545a0" Jul 13 22:00:23.615133 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:23.615114 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-87w2d"] Jul 13 22:00:23.615241 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:23.615229 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87w2d" Jul 13 22:00:23.615322 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:23.615303 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87w2d" podUID="55ac7566-430e-4e45-ab69-fe943cfb4c44" Jul 13 22:00:23.616190 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:23.616166 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5bkm9"] Jul 13 22:00:23.694411 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:23.694339 2555 generic.go:356] "Generic (PLEG): container finished" podID="d94f40ee-a429-4bf2-b7cb-c436922cdedc" containerID="7a3e1f239a8eecf1c4cef6ed7ad1e99b98925d4f1f7c2226c4918a7086f7956f" exitCode=0 Jul 13 22:00:23.694536 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:23.694423 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6xnxl" event={"ID":"d94f40ee-a429-4bf2-b7cb-c436922cdedc","Type":"ContainerDied","Data":"7a3e1f239a8eecf1c4cef6ed7ad1e99b98925d4f1f7c2226c4918a7086f7956f"} Jul 13 22:00:23.694536 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:23.694468 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5bkm9" Jul 13 22:00:23.694974 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:23.694937 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5bkm9" podUID="56086743-e831-415a-a2c3-6982ef099751" Jul 13 22:00:25.554009 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:25.553801 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5bkm9" Jul 13 22:00:25.554478 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:25.553834 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87w2d" Jul 13 22:00:25.554478 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:25.553855 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rg8tn" Jul 13 22:00:25.554478 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:25.554105 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5bkm9" podUID="56086743-e831-415a-a2c3-6982ef099751" Jul 13 22:00:25.557355 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:25.554727 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87w2d" podUID="55ac7566-430e-4e45-ab69-fe943cfb4c44" Jul 13 22:00:25.557355 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:25.555018 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rg8tn" podUID="d1b9ac92-b63a-4180-9293-86c6b0c545a0" Jul 13 22:00:25.701608 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:25.701526 2555 generic.go:356] "Generic (PLEG): container finished" podID="d94f40ee-a429-4bf2-b7cb-c436922cdedc" containerID="6fb9684dca57bb971646f3c14510cb048968fda80da04f3855504411d93aff9f" exitCode=0 Jul 13 22:00:25.701608 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:25.701573 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6xnxl" event={"ID":"d94f40ee-a429-4bf2-b7cb-c436922cdedc","Type":"ContainerDied","Data":"6fb9684dca57bb971646f3c14510cb048968fda80da04f3855504411d93aff9f"} Jul 13 22:00:27.553428 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:27.553393 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rg8tn" Jul 13 22:00:27.553428 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:27.553415 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87w2d" Jul 13 22:00:27.553906 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:27.553438 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5bkm9" Jul 13 22:00:27.553906 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:27.553541 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rg8tn" podUID="d1b9ac92-b63a-4180-9293-86c6b0c545a0" Jul 13 22:00:27.553906 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:27.553639 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87w2d" podUID="55ac7566-430e-4e45-ab69-fe943cfb4c44" Jul 13 22:00:27.553906 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:27.553705 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5bkm9" podUID="56086743-e831-415a-a2c3-6982ef099751" Jul 13 22:00:28.290685 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:28.290645 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56086743-e831-415a-a2c3-6982ef099751-original-pull-secret\") pod \"global-pull-secret-syncer-5bkm9\" (UID: \"56086743-e831-415a-a2c3-6982ef099751\") " pod="kube-system/global-pull-secret-syncer-5bkm9" Jul 13 22:00:28.290872 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:28.290721 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55ac7566-430e-4e45-ab69-fe943cfb4c44-metrics-certs\") pod \"network-metrics-daemon-87w2d\" (UID: \"55ac7566-430e-4e45-ab69-fe943cfb4c44\") " pod="openshift-multus/network-metrics-daemon-87w2d" Jul 13 22:00:28.290872 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:28.290818 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 22:00:28.290872 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:28.290847 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 22:00:28.290991 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:28.290891 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56086743-e831-415a-a2c3-6982ef099751-original-pull-secret podName:56086743-e831-415a-a2c3-6982ef099751 nodeName:}" failed. No retries permitted until 2026-07-13 22:01:00.290873162 +0000 UTC m=+66.284627447 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/56086743-e831-415a-a2c3-6982ef099751-original-pull-secret") pod "global-pull-secret-syncer-5bkm9" (UID: "56086743-e831-415a-a2c3-6982ef099751") : object "kube-system"/"original-pull-secret" not registered Jul 13 22:00:28.290991 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:28.290908 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55ac7566-430e-4e45-ab69-fe943cfb4c44-metrics-certs podName:55ac7566-430e-4e45-ab69-fe943cfb4c44 nodeName:}" failed. No retries permitted until 2026-07-13 22:01:00.290900948 +0000 UTC m=+66.284655218 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55ac7566-430e-4e45-ab69-fe943cfb4c44-metrics-certs") pod "network-metrics-daemon-87w2d" (UID: "55ac7566-430e-4e45-ab69-fe943cfb4c44") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 22:00:28.391971 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:28.391941 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8h4k\" (UniqueName: \"kubernetes.io/projected/d1b9ac92-b63a-4180-9293-86c6b0c545a0-kube-api-access-h8h4k\") pod \"network-check-target-rg8tn\" (UID: \"d1b9ac92-b63a-4180-9293-86c6b0c545a0\") " pod="openshift-network-diagnostics/network-check-target-rg8tn" Jul 13 22:00:28.392192 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:28.392113 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 22:00:28.392192 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:28.392134 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 22:00:28.392192 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:28.392148 2555 projected.go:196] Error preparing data for projected volume kube-api-access-h8h4k for pod openshift-network-diagnostics/network-check-target-rg8tn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 22:00:28.392482 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:28.392216 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1b9ac92-b63a-4180-9293-86c6b0c545a0-kube-api-access-h8h4k podName:d1b9ac92-b63a-4180-9293-86c6b0c545a0 nodeName:}" failed. No retries permitted until 2026-07-13 22:01:00.392186443 +0000 UTC m=+66.385940713 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-h8h4k" (UniqueName: "kubernetes.io/projected/d1b9ac92-b63a-4180-9293-86c6b0c545a0-kube-api-access-h8h4k") pod "network-check-target-rg8tn" (UID: "d1b9ac92-b63a-4180-9293-86c6b0c545a0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 22:00:29.315590 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:29.315508 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-234.ec2.internal" event="NodeReady" Jul 13 22:00:29.316106 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:29.315661 2555 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 13 22:00:29.370434 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:29.370399 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-7hh44"] Jul 13 22:00:29.376840 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:29.376807 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-kr6fz"] Jul 13 22:00:29.377014 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:29.376972 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7hh44" Jul 13 22:00:29.378910 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:29.378890 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kr6fz" Jul 13 22:00:29.380355 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:29.380324 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 13 22:00:29.380515 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:29.380498 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-89pgf\"" Jul 13 22:00:29.380729 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:29.380713 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 13 22:00:29.380729 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:29.380722 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 13 22:00:29.381542 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:29.381523 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 13 22:00:29.382231 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:29.382192 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 13 22:00:29.382334 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:29.382280 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xvmjd\"" Jul 13 22:00:29.382334 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:29.382310 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 13 22:00:29.382641 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:29.382622 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 13 22:00:29.396189 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:29.396167 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7hh44"] Jul 13 22:00:29.396984 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:29.396962 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kr6fz"] Jul 13 22:00:29.400831 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:29.400811 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-pqjkz"] Jul 13 22:00:29.402753 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:29.402738 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pqjkz" Jul 13 22:00:29.405646 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:29.405624 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 13 22:00:29.405735 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:29.405624 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 13 22:00:29.405735 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:29.405701 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lvbhm\"" Jul 13 22:00:29.414286 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:29.414264 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pqjkz"] Jul 13 22:00:29.500827 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:29.500798 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9d5db581-bfb8-4fa9-a027-80f98597e1b0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kr6fz\" (UID: \"9d5db581-bfb8-4fa9-a027-80f98597e1b0\") " pod="openshift-insights/insights-runtime-extractor-kr6fz" Jul 13 22:00:29.500827 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:29.500833 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bbqz\" (UniqueName: \"kubernetes.io/projected/9d5db581-bfb8-4fa9-a027-80f98597e1b0-kube-api-access-2bbqz\") pod \"insights-runtime-extractor-kr6fz\" (UID: \"9d5db581-bfb8-4fa9-a027-80f98597e1b0\") " pod="openshift-insights/insights-runtime-extractor-kr6fz" Jul 13 22:00:29.501036 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:29.500863 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9d5db581-bfb8-4fa9-a027-80f98597e1b0-data-volume\") pod \"insights-runtime-extractor-kr6fz\" (UID: \"9d5db581-bfb8-4fa9-a027-80f98597e1b0\") " pod="openshift-insights/insights-runtime-extractor-kr6fz" Jul 13 22:00:29.501036 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:29.500921 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/54f54657-10eb-47ae-ba8f-ec453e686d04-metrics-tls\") pod \"dns-default-pqjkz\" (UID: \"54f54657-10eb-47ae-ba8f-ec453e686d04\") " pod="openshift-dns/dns-default-pqjkz" Jul 13 22:00:29.501036 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:29.500956 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlfdn\" (UniqueName: \"kubernetes.io/projected/42289584-c895-4ac8-a13f-d718fa245599-kube-api-access-vlfdn\") pod \"ingress-canary-7hh44\" (UID: \"42289584-c895-4ac8-a13f-d718fa245599\") " pod="openshift-ingress-canary/ingress-canary-7hh44" Jul 13 22:00:29.501036 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:29.500998 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/42289584-c895-4ac8-a13f-d718fa245599-cert\") pod \"ingress-canary-7hh44\" (UID: \"42289584-c895-4ac8-a13f-d718fa245599\") " pod="openshift-ingress-canary/ingress-canary-7hh44" Jul 13 22:00:29.501192 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:29.501053 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/54f54657-10eb-47ae-ba8f-ec453e686d04-config-volume\") pod \"dns-default-pqjkz\" (UID: \"54f54657-10eb-47ae-ba8f-ec453e686d04\") " pod="openshift-dns/dns-default-pqjkz" Jul 13 22:00:29.501192 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:29.501100 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9d5db581-bfb8-4fa9-a027-80f98597e1b0-crio-socket\") pod \"insights-runtime-extractor-kr6fz\" (UID: \"9d5db581-bfb8-4fa9-a027-80f98597e1b0\") " pod="openshift-insights/insights-runtime-extractor-kr6fz" Jul 13 22:00:29.501192 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:29.501130 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d5db581-bfb8-4fa9-a027-80f98597e1b0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kr6fz\" (UID: \"9d5db581-bfb8-4fa9-a027-80f98597e1b0\") " pod="openshift-insights/insights-runtime-extractor-kr6fz" Jul 13 22:00:29.501192 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:29.501157 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54f54657-10eb-47ae-ba8f-ec453e686d04-tmp-dir\") pod \"dns-default-pqjkz\" (UID: \"54f54657-10eb-47ae-ba8f-ec453e686d04\") " pod="openshift-dns/dns-default-pqjkz" Jul 13 22:00:29.501192 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:29.501182 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fkvr\" (UniqueName: \"kubernetes.io/projected/54f54657-10eb-47ae-ba8f-ec453e686d04-kube-api-access-9fkvr\") pod \"dns-default-pqjkz\" (UID: \"54f54657-10eb-47ae-ba8f-ec453e686d04\") " pod="openshift-dns/dns-default-pqjkz" Jul 13 22:00:29.554118 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:29.554088 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rg8tn" Jul 13 22:00:29.554557 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:29.554539 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87w2d" Jul 13 22:00:29.554948 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:29.554934 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5bkm9" Jul 13 22:00:29.557403 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:29.557382 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 13 22:00:29.557531 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:29.557414 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 13 22:00:29.557622 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:29.557598 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 13 22:00:29.557748 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:29.557696 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gmvbg\"" Jul 13 22:00:29.557957 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:29.557940 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 13 22:00:29.558047 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:29.557988 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-b6tsw\"" Jul 13 22:00:29.602528 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:29.602505 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9d5db581-bfb8-4fa9-a027-80f98597e1b0-crio-socket\") pod \"insights-runtime-extractor-kr6fz\" (UID: \"9d5db581-bfb8-4fa9-a027-80f98597e1b0\") " pod="openshift-insights/insights-runtime-extractor-kr6fz" Jul 13 22:00:29.602639 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:29.602544 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d5db581-bfb8-4fa9-a027-80f98597e1b0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kr6fz\" (UID: \"9d5db581-bfb8-4fa9-a027-80f98597e1b0\") " pod="openshift-insights/insights-runtime-extractor-kr6fz" Jul 13 22:00:29.602639 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:29.602563 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54f54657-10eb-47ae-ba8f-ec453e686d04-tmp-dir\") pod \"dns-default-pqjkz\" (UID: \"54f54657-10eb-47ae-ba8f-ec453e686d04\") " pod="openshift-dns/dns-default-pqjkz" Jul 13 22:00:29.602742 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:29.602679 2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 13 22:00:29.602792 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:29.602757 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d5db581-bfb8-4fa9-a027-80f98597e1b0-insights-runtime-extractor-tls podName:9d5db581-bfb8-4fa9-a027-80f98597e1b0 nodeName:}" failed. No retries permitted until 2026-07-13 22:00:30.102733803 +0000 UTC m=+36.096488078 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9d5db581-bfb8-4fa9-a027-80f98597e1b0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kr6fz" (UID: "9d5db581-bfb8-4fa9-a027-80f98597e1b0") : secret "insights-runtime-extractor-tls" not found Jul 13 22:00:29.602857 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:29.602783 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9d5db581-bfb8-4fa9-a027-80f98597e1b0-crio-socket\") pod \"insights-runtime-extractor-kr6fz\" (UID: \"9d5db581-bfb8-4fa9-a027-80f98597e1b0\") " pod="openshift-insights/insights-runtime-extractor-kr6fz" Jul 13 22:00:29.602857 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:29.602835 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54f54657-10eb-47ae-ba8f-ec453e686d04-tmp-dir\") pod \"dns-default-pqjkz\" (UID: \"54f54657-10eb-47ae-ba8f-ec453e686d04\") " pod="openshift-dns/dns-default-pqjkz" Jul 13 22:00:29.602857 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:29.602835 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fkvr\" (UniqueName: \"kubernetes.io/projected/54f54657-10eb-47ae-ba8f-ec453e686d04-kube-api-access-9fkvr\") pod \"dns-default-pqjkz\" (UID: \"54f54657-10eb-47ae-ba8f-ec453e686d04\") " pod="openshift-dns/dns-default-pqjkz" Jul 13 22:00:29.603008 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:29.602877 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9d5db581-bfb8-4fa9-a027-80f98597e1b0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kr6fz\" (UID: \"9d5db581-bfb8-4fa9-a027-80f98597e1b0\") " pod="openshift-insights/insights-runtime-extractor-kr6fz" Jul 13 22:00:29.603008 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:29.602899 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bbqz\" (UniqueName: \"kubernetes.io/projected/9d5db581-bfb8-4fa9-a027-80f98597e1b0-kube-api-access-2bbqz\") pod \"insights-runtime-extractor-kr6fz\" (UID: \"9d5db581-bfb8-4fa9-a027-80f98597e1b0\") " pod="openshift-insights/insights-runtime-extractor-kr6fz" Jul 13 22:00:29.603008 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:29.602942 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9d5db581-bfb8-4fa9-a027-80f98597e1b0-data-volume\") pod \"insights-runtime-extractor-kr6fz\" (UID: \"9d5db581-bfb8-4fa9-a027-80f98597e1b0\") " pod="openshift-insights/insights-runtime-extractor-kr6fz" Jul 13 22:00:29.603008 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:29.602966 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/54f54657-10eb-47ae-ba8f-ec453e686d04-metrics-tls\") pod \"dns-default-pqjkz\" (UID: \"54f54657-10eb-47ae-ba8f-ec453e686d04\") " pod="openshift-dns/dns-default-pqjkz" Jul 13 22:00:29.603008 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:29.602992 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlfdn\" (UniqueName: \"kubernetes.io/projected/42289584-c895-4ac8-a13f-d718fa245599-kube-api-access-vlfdn\") pod \"ingress-canary-7hh44\" (UID: \"42289584-c895-4ac8-a13f-d718fa245599\") " pod="openshift-ingress-canary/ingress-canary-7hh44" Jul 13 22:00:29.603350 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:29.603011 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/42289584-c895-4ac8-a13f-d718fa245599-cert\") pod \"ingress-canary-7hh44\" (UID: \"42289584-c895-4ac8-a13f-d718fa245599\") " pod="openshift-ingress-canary/ingress-canary-7hh44" Jul 13 22:00:29.603350 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:29.603168 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 13 22:00:29.603350 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:29.603252 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54f54657-10eb-47ae-ba8f-ec453e686d04-metrics-tls podName:54f54657-10eb-47ae-ba8f-ec453e686d04 nodeName:}" failed. No retries permitted until 2026-07-13 22:00:30.103234918 +0000 UTC m=+36.096989202 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/54f54657-10eb-47ae-ba8f-ec453e686d04-metrics-tls") pod "dns-default-pqjkz" (UID: "54f54657-10eb-47ae-ba8f-ec453e686d04") : secret "dns-default-metrics-tls" not found Jul 13 22:00:29.603350 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:29.603263 2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 13 22:00:29.603350 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:29.603314 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9d5db581-bfb8-4fa9-a027-80f98597e1b0-kube-rbac-proxy-cm podName:9d5db581-bfb8-4fa9-a027-80f98597e1b0 nodeName:}" failed. No retries permitted until 2026-07-13 22:00:30.103299338 +0000 UTC m=+36.097053612 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9d5db581-bfb8-4fa9-a027-80f98597e1b0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kr6fz" (UID: "9d5db581-bfb8-4fa9-a027-80f98597e1b0") : configmap "kube-rbac-proxy" not found Jul 13 22:00:29.603350 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:29.603339 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/54f54657-10eb-47ae-ba8f-ec453e686d04-config-volume\") pod \"dns-default-pqjkz\" (UID: \"54f54657-10eb-47ae-ba8f-ec453e686d04\") " pod="openshift-dns/dns-default-pqjkz" Jul 13 22:00:29.603637 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:29.603570 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9d5db581-bfb8-4fa9-a027-80f98597e1b0-data-volume\") pod \"insights-runtime-extractor-kr6fz\" (UID: \"9d5db581-bfb8-4fa9-a027-80f98597e1b0\") " pod="openshift-insights/insights-runtime-extractor-kr6fz" Jul 13 22:00:29.603907 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:29.603859 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/54f54657-10eb-47ae-ba8f-ec453e686d04-config-volume\") pod \"dns-default-pqjkz\" (UID: \"54f54657-10eb-47ae-ba8f-ec453e686d04\") " pod="openshift-dns/dns-default-pqjkz" Jul 13 22:00:29.607652 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:29.607634 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/42289584-c895-4ac8-a13f-d718fa245599-cert\") pod \"ingress-canary-7hh44\" (UID: \"42289584-c895-4ac8-a13f-d718fa245599\") " pod="openshift-ingress-canary/ingress-canary-7hh44" Jul 13 22:00:29.610992 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:29.610947 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fkvr\" (UniqueName: \"kubernetes.io/projected/54f54657-10eb-47ae-ba8f-ec453e686d04-kube-api-access-9fkvr\") pod \"dns-default-pqjkz\" (UID: \"54f54657-10eb-47ae-ba8f-ec453e686d04\") " pod="openshift-dns/dns-default-pqjkz" Jul 13 22:00:29.614417 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:29.614395 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bbqz\" (UniqueName: \"kubernetes.io/projected/9d5db581-bfb8-4fa9-a027-80f98597e1b0-kube-api-access-2bbqz\") pod \"insights-runtime-extractor-kr6fz\" (UID: \"9d5db581-bfb8-4fa9-a027-80f98597e1b0\") " pod="openshift-insights/insights-runtime-extractor-kr6fz" Jul 13 22:00:29.615629 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:29.615608 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlfdn\" (UniqueName: \"kubernetes.io/projected/42289584-c895-4ac8-a13f-d718fa245599-kube-api-access-vlfdn\") pod \"ingress-canary-7hh44\" (UID: \"42289584-c895-4ac8-a13f-d718fa245599\") " pod="openshift-ingress-canary/ingress-canary-7hh44" Jul 13 22:00:29.689652 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:29.689616 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7hh44" Jul 13 22:00:30.108247 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:30.108160 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/54f54657-10eb-47ae-ba8f-ec453e686d04-metrics-tls\") pod \"dns-default-pqjkz\" (UID: \"54f54657-10eb-47ae-ba8f-ec453e686d04\") " pod="openshift-dns/dns-default-pqjkz" Jul 13 22:00:30.108365 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:30.108289 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d5db581-bfb8-4fa9-a027-80f98597e1b0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kr6fz\" (UID: \"9d5db581-bfb8-4fa9-a027-80f98597e1b0\") " pod="openshift-insights/insights-runtime-extractor-kr6fz" Jul 13 22:00:30.108365 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:30.108336 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9d5db581-bfb8-4fa9-a027-80f98597e1b0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kr6fz\" (UID: \"9d5db581-bfb8-4fa9-a027-80f98597e1b0\") " pod="openshift-insights/insights-runtime-extractor-kr6fz" Jul 13 22:00:30.108899 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:30.108874 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9d5db581-bfb8-4fa9-a027-80f98597e1b0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kr6fz\" (UID: \"9d5db581-bfb8-4fa9-a027-80f98597e1b0\") " pod="openshift-insights/insights-runtime-extractor-kr6fz" Jul 13 22:00:30.109012 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:30.108996 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 13 22:00:30.109074 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:30.109056 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54f54657-10eb-47ae-ba8f-ec453e686d04-metrics-tls podName:54f54657-10eb-47ae-ba8f-ec453e686d04 nodeName:}" failed. No retries permitted until 2026-07-13 22:00:31.109035409 +0000 UTC m=+37.102789694 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/54f54657-10eb-47ae-ba8f-ec453e686d04-metrics-tls") pod "dns-default-pqjkz" (UID: "54f54657-10eb-47ae-ba8f-ec453e686d04") : secret "dns-default-metrics-tls" not found Jul 13 22:00:30.109142 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:30.109109 2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 13 22:00:30.109192 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:30.109146 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d5db581-bfb8-4fa9-a027-80f98597e1b0-insights-runtime-extractor-tls podName:9d5db581-bfb8-4fa9-a027-80f98597e1b0 nodeName:}" failed. No retries permitted until 2026-07-13 22:00:31.109134229 +0000 UTC m=+37.102888505 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9d5db581-bfb8-4fa9-a027-80f98597e1b0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kr6fz" (UID: "9d5db581-bfb8-4fa9-a027-80f98597e1b0") : secret "insights-runtime-extractor-tls" not found Jul 13 22:00:30.190795 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:30.190650 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7hh44"] Jul 13 22:00:30.193829 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:00:30.193802 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42289584_c895_4ac8_a13f_d718fa245599.slice/crio-bd8fdc0053141e1d00f97d522332c020e20f0b95d8e51e0f1e392511fd214195 WatchSource:0}: Error finding container bd8fdc0053141e1d00f97d522332c020e20f0b95d8e51e0f1e392511fd214195: Status 404 returned error can't find the container with id bd8fdc0053141e1d00f97d522332c020e20f0b95d8e51e0f1e392511fd214195 Jul 13 22:00:30.716221 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:30.716170 2555 generic.go:356] "Generic (PLEG): container finished" podID="d94f40ee-a429-4bf2-b7cb-c436922cdedc" containerID="1917c6d9c4824d3985e991b4a38af5f2fb834a37562c13631cb32b0d0c8724ad" exitCode=0 Jul 13 22:00:30.716796 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:30.716226 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6xnxl" event={"ID":"d94f40ee-a429-4bf2-b7cb-c436922cdedc","Type":"ContainerDied","Data":"1917c6d9c4824d3985e991b4a38af5f2fb834a37562c13631cb32b0d0c8724ad"} Jul 13 22:00:30.717639 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:30.717612 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7hh44" event={"ID":"42289584-c895-4ac8-a13f-d718fa245599","Type":"ContainerStarted","Data":"bd8fdc0053141e1d00f97d522332c020e20f0b95d8e51e0f1e392511fd214195"} Jul 13 22:00:30.860876 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:30.860845 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-ckmk5"] Jul 13 22:00:30.862549 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:30.862528 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-ckmk5" Jul 13 22:00:30.865056 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:30.864982 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 13 22:00:30.865056 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:30.865010 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 13 22:00:30.865056 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:30.865048 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-bs84w\"" Jul 13 22:00:30.865311 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:30.865059 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 13 22:00:30.865311 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:30.865076 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 13 22:00:30.873736 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:30.873711 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-ckmk5"] Jul 13 22:00:30.954143 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:30.954118 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2znr7_9985a1e9-5e38-4bea-9745-988c4524fc21/dns-node-resolver/0.log" Jul 13 22:00:31.016530 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:31.016440 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/60d6b0ad-83fe-4182-91da-6f7a1add9eb5-signing-key\") pod \"service-ca-79f68b9f4f-ckmk5\" (UID: \"60d6b0ad-83fe-4182-91da-6f7a1add9eb5\") " pod="openshift-service-ca/service-ca-79f68b9f4f-ckmk5" Jul 13 22:00:31.016530 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:31.016482 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppmvx\" (UniqueName: \"kubernetes.io/projected/60d6b0ad-83fe-4182-91da-6f7a1add9eb5-kube-api-access-ppmvx\") pod \"service-ca-79f68b9f4f-ckmk5\" (UID: \"60d6b0ad-83fe-4182-91da-6f7a1add9eb5\") " pod="openshift-service-ca/service-ca-79f68b9f4f-ckmk5" Jul 13 22:00:31.016726 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:31.016656 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/60d6b0ad-83fe-4182-91da-6f7a1add9eb5-signing-cabundle\") pod \"service-ca-79f68b9f4f-ckmk5\" (UID: \"60d6b0ad-83fe-4182-91da-6f7a1add9eb5\") " pod="openshift-service-ca/service-ca-79f68b9f4f-ckmk5" Jul 13 22:00:31.117311 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:31.117276 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/54f54657-10eb-47ae-ba8f-ec453e686d04-metrics-tls\") pod \"dns-default-pqjkz\" (UID: \"54f54657-10eb-47ae-ba8f-ec453e686d04\") " pod="openshift-dns/dns-default-pqjkz" Jul 13 22:00:31.117526 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:31.117322 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/60d6b0ad-83fe-4182-91da-6f7a1add9eb5-signing-key\") pod \"service-ca-79f68b9f4f-ckmk5\" (UID: \"60d6b0ad-83fe-4182-91da-6f7a1add9eb5\") " pod="openshift-service-ca/service-ca-79f68b9f4f-ckmk5" Jul 13 22:00:31.117526 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:31.117352 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppmvx\" (UniqueName: \"kubernetes.io/projected/60d6b0ad-83fe-4182-91da-6f7a1add9eb5-kube-api-access-ppmvx\") pod \"service-ca-79f68b9f4f-ckmk5\" (UID: \"60d6b0ad-83fe-4182-91da-6f7a1add9eb5\") " pod="openshift-service-ca/service-ca-79f68b9f4f-ckmk5" Jul 13 22:00:31.117526 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:31.117435 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d5db581-bfb8-4fa9-a027-80f98597e1b0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kr6fz\" (UID: \"9d5db581-bfb8-4fa9-a027-80f98597e1b0\") " pod="openshift-insights/insights-runtime-extractor-kr6fz" Jul 13 22:00:31.117526 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:31.117440 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 13 22:00:31.117526 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:31.117472 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/60d6b0ad-83fe-4182-91da-6f7a1add9eb5-signing-cabundle\") pod \"service-ca-79f68b9f4f-ckmk5\" (UID: \"60d6b0ad-83fe-4182-91da-6f7a1add9eb5\") " pod="openshift-service-ca/service-ca-79f68b9f4f-ckmk5" Jul 13 22:00:31.117526 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:31.117520 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54f54657-10eb-47ae-ba8f-ec453e686d04-metrics-tls podName:54f54657-10eb-47ae-ba8f-ec453e686d04 nodeName:}" failed. No retries permitted until 2026-07-13 22:00:33.117495962 +0000 UTC m=+39.111250251 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/54f54657-10eb-47ae-ba8f-ec453e686d04-metrics-tls") pod "dns-default-pqjkz" (UID: "54f54657-10eb-47ae-ba8f-ec453e686d04") : secret "dns-default-metrics-tls" not found Jul 13 22:00:31.117831 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:31.117747 2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 13 22:00:31.117831 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:31.117793 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d5db581-bfb8-4fa9-a027-80f98597e1b0-insights-runtime-extractor-tls podName:9d5db581-bfb8-4fa9-a027-80f98597e1b0 nodeName:}" failed. No retries permitted until 2026-07-13 22:00:33.11778133 +0000 UTC m=+39.111535602 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9d5db581-bfb8-4fa9-a027-80f98597e1b0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kr6fz" (UID: "9d5db581-bfb8-4fa9-a027-80f98597e1b0") : secret "insights-runtime-extractor-tls" not found Jul 13 22:00:31.118249 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:31.118227 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/60d6b0ad-83fe-4182-91da-6f7a1add9eb5-signing-cabundle\") pod \"service-ca-79f68b9f4f-ckmk5\" (UID: \"60d6b0ad-83fe-4182-91da-6f7a1add9eb5\") " pod="openshift-service-ca/service-ca-79f68b9f4f-ckmk5" Jul 13 22:00:31.121031 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:31.121001 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/60d6b0ad-83fe-4182-91da-6f7a1add9eb5-signing-key\") pod \"service-ca-79f68b9f4f-ckmk5\" (UID: \"60d6b0ad-83fe-4182-91da-6f7a1add9eb5\") " pod="openshift-service-ca/service-ca-79f68b9f4f-ckmk5" Jul 13 22:00:31.126898 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:31.126874 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppmvx\" (UniqueName: \"kubernetes.io/projected/60d6b0ad-83fe-4182-91da-6f7a1add9eb5-kube-api-access-ppmvx\") pod \"service-ca-79f68b9f4f-ckmk5\" (UID: \"60d6b0ad-83fe-4182-91da-6f7a1add9eb5\") " pod="openshift-service-ca/service-ca-79f68b9f4f-ckmk5" Jul 13 22:00:31.191035 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:31.191001 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-ckmk5" Jul 13 22:00:31.330994 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:31.330966 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-ckmk5"] Jul 13 22:00:31.335975 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:00:31.335948 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60d6b0ad_83fe_4182_91da_6f7a1add9eb5.slice/crio-966af6e8e437e916bc73c3dd123aad184389563a98231a9a7f40085d79275d38 WatchSource:0}: Error finding container 966af6e8e437e916bc73c3dd123aad184389563a98231a9a7f40085d79275d38: Status 404 returned error can't find the container with id 966af6e8e437e916bc73c3dd123aad184389563a98231a9a7f40085d79275d38 Jul 13 22:00:31.721245 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:31.721139 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-ckmk5" event={"ID":"60d6b0ad-83fe-4182-91da-6f7a1add9eb5","Type":"ContainerStarted","Data":"966af6e8e437e916bc73c3dd123aad184389563a98231a9a7f40085d79275d38"} Jul 13 22:00:31.724851 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:31.724822 2555 generic.go:356] "Generic (PLEG): container finished" podID="d94f40ee-a429-4bf2-b7cb-c436922cdedc" containerID="0e9d2f28f951ad482ccce5bcc23c4074482e3af7cd4a4c315e7ad248e44f81a5" exitCode=0 Jul 13 22:00:31.724974 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:31.724864 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6xnxl" event={"ID":"d94f40ee-a429-4bf2-b7cb-c436922cdedc","Type":"ContainerDied","Data":"0e9d2f28f951ad482ccce5bcc23c4074482e3af7cd4a4c315e7ad248e44f81a5"} Jul 13 22:00:31.939105 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:31.939066 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6nq95_54550ef2-07f8-4e03-b898-c5d7536cc61e/node-ca/0.log" Jul 13 22:00:32.732772 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:32.732464 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6xnxl" event={"ID":"d94f40ee-a429-4bf2-b7cb-c436922cdedc","Type":"ContainerStarted","Data":"b7061c01ce55c1e288bdb1d657fcb62f0749f699c25d9e05b602e6a183700c39"} Jul 13 22:00:32.734645 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:32.734618 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7hh44" event={"ID":"42289584-c895-4ac8-a13f-d718fa245599","Type":"ContainerStarted","Data":"d4c68e13b4ae5604fcb8ec186b27eafe547890424852c8740752b2501e232e4a"} Jul 13 22:00:32.756590 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:32.756541 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6xnxl" podStartSLOduration=8.485330168 podStartE2EDuration="38.756529643s" totalImagesPullingTime="30.271199475s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 21:59:54 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:00:32.754989765 +0000 UTC m=+38.748744084" watchObservedRunningTime="2026-07-13 22:00:32.756529643 +0000 UTC m=+38.750283936" Jul 13 22:00:33.139259 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:33.139221 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d5db581-bfb8-4fa9-a027-80f98597e1b0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kr6fz\" (UID: \"9d5db581-bfb8-4fa9-a027-80f98597e1b0\") " pod="openshift-insights/insights-runtime-extractor-kr6fz" Jul 13 22:00:33.139397 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:33.139285 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/54f54657-10eb-47ae-ba8f-ec453e686d04-metrics-tls\") pod \"dns-default-pqjkz\" (UID: \"54f54657-10eb-47ae-ba8f-ec453e686d04\") " pod="openshift-dns/dns-default-pqjkz" Jul 13 22:00:33.139397 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:33.139386 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 13 22:00:33.139397 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:33.139386 2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 13 22:00:33.139495 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:33.139451 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d5db581-bfb8-4fa9-a027-80f98597e1b0-insights-runtime-extractor-tls podName:9d5db581-bfb8-4fa9-a027-80f98597e1b0 nodeName:}" failed. No retries permitted until 2026-07-13 22:00:37.139434513 +0000 UTC m=+43.133188784 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9d5db581-bfb8-4fa9-a027-80f98597e1b0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kr6fz" (UID: "9d5db581-bfb8-4fa9-a027-80f98597e1b0") : secret "insights-runtime-extractor-tls" not found Jul 13 22:00:33.139495 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:33.139466 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54f54657-10eb-47ae-ba8f-ec453e686d04-metrics-tls podName:54f54657-10eb-47ae-ba8f-ec453e686d04 nodeName:}" failed. No retries permitted until 2026-07-13 22:00:37.139459532 +0000 UTC m=+43.133213803 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/54f54657-10eb-47ae-ba8f-ec453e686d04-metrics-tls") pod "dns-default-pqjkz" (UID: "54f54657-10eb-47ae-ba8f-ec453e686d04") : secret "dns-default-metrics-tls" not found Jul 13 22:00:34.741102 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:34.741071 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-ckmk5" event={"ID":"60d6b0ad-83fe-4182-91da-6f7a1add9eb5","Type":"ContainerStarted","Data":"83fdc2c054e81b688f8f94066710a061542112b6700925016a9f227e083df1e7"} Jul 13 22:00:34.757079 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:34.757018 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-ckmk5" podStartSLOduration=1.431966145 podStartE2EDuration="4.756999667s" totalImagesPullingTime="3.325033522s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:00:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:00:34.756616407 +0000 UTC m=+40.750370715" watchObservedRunningTime="2026-07-13 22:00:34.756999667 +0000 UTC m=+40.750753961" Jul 13 22:00:34.757353 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:34.757330 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-7hh44" podStartSLOduration=3.5236397779999997 podStartE2EDuration="5.757322155s" totalImagesPullingTime="2.233682377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:00:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:00:32.772963289 +0000 UTC m=+38.766717604" watchObservedRunningTime="2026-07-13 22:00:34.757322155 +0000 UTC m=+40.751076456" Jul 13 22:00:37.170459 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:37.170418 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/54f54657-10eb-47ae-ba8f-ec453e686d04-metrics-tls\") pod \"dns-default-pqjkz\" (UID: \"54f54657-10eb-47ae-ba8f-ec453e686d04\") " pod="openshift-dns/dns-default-pqjkz" Jul 13 22:00:37.170890 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:37.170545 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 13 22:00:37.170890 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:37.170610 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54f54657-10eb-47ae-ba8f-ec453e686d04-metrics-tls podName:54f54657-10eb-47ae-ba8f-ec453e686d04 nodeName:}" failed. No retries permitted until 2026-07-13 22:00:45.170593414 +0000 UTC m=+51.164347686 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/54f54657-10eb-47ae-ba8f-ec453e686d04-metrics-tls") pod "dns-default-pqjkz" (UID: "54f54657-10eb-47ae-ba8f-ec453e686d04") : secret "dns-default-metrics-tls" not found Jul 13 22:00:37.170890 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:37.170550 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d5db581-bfb8-4fa9-a027-80f98597e1b0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kr6fz\" (UID: \"9d5db581-bfb8-4fa9-a027-80f98597e1b0\") " pod="openshift-insights/insights-runtime-extractor-kr6fz" Jul 13 22:00:37.170890 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:37.170645 2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 13 22:00:37.170890 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:00:37.170715 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d5db581-bfb8-4fa9-a027-80f98597e1b0-insights-runtime-extractor-tls podName:9d5db581-bfb8-4fa9-a027-80f98597e1b0 nodeName:}" failed. No retries permitted until 2026-07-13 22:00:45.170698544 +0000 UTC m=+51.164452820 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9d5db581-bfb8-4fa9-a027-80f98597e1b0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kr6fz" (UID: "9d5db581-bfb8-4fa9-a027-80f98597e1b0") : secret "insights-runtime-extractor-tls" not found Jul 13 22:00:45.232635 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:45.232596 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/54f54657-10eb-47ae-ba8f-ec453e686d04-metrics-tls\") pod \"dns-default-pqjkz\" (UID: \"54f54657-10eb-47ae-ba8f-ec453e686d04\") " pod="openshift-dns/dns-default-pqjkz" Jul 13 22:00:45.233225 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:45.232690 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d5db581-bfb8-4fa9-a027-80f98597e1b0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kr6fz\" (UID: \"9d5db581-bfb8-4fa9-a027-80f98597e1b0\") " pod="openshift-insights/insights-runtime-extractor-kr6fz" Jul 13 22:00:45.236100 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:45.236079 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d5db581-bfb8-4fa9-a027-80f98597e1b0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kr6fz\" (UID: \"9d5db581-bfb8-4fa9-a027-80f98597e1b0\") " pod="openshift-insights/insights-runtime-extractor-kr6fz" Jul 13 22:00:45.236186 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:45.236111 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/54f54657-10eb-47ae-ba8f-ec453e686d04-metrics-tls\") pod \"dns-default-pqjkz\" (UID: \"54f54657-10eb-47ae-ba8f-ec453e686d04\") " pod="openshift-dns/dns-default-pqjkz" Jul 13 22:00:45.297090 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:45.297063 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kr6fz" Jul 13 22:00:45.312766 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:45.312741 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pqjkz" Jul 13 22:00:45.484856 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:45.484778 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kr6fz"] Jul 13 22:00:45.487783 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:00:45.487756 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d5db581_bfb8_4fa9_a027_80f98597e1b0.slice/crio-b27ca5b256a356691f297b87c5107876fa11635044b1795f0102faa3312aaf62 WatchSource:0}: Error finding container b27ca5b256a356691f297b87c5107876fa11635044b1795f0102faa3312aaf62: Status 404 returned error can't find the container with id b27ca5b256a356691f297b87c5107876fa11635044b1795f0102faa3312aaf62 Jul 13 22:00:45.489304 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:45.489280 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pqjkz"] Jul 13 22:00:45.492741 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:00:45.492713 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54f54657_10eb_47ae_ba8f_ec453e686d04.slice/crio-b29cee968700f47f015ec499de6e6c8330ec609a94c59a04fc4bf82c4fced795 WatchSource:0}: Error finding container b29cee968700f47f015ec499de6e6c8330ec609a94c59a04fc4bf82c4fced795: Status 404 returned error can't find the container with id b29cee968700f47f015ec499de6e6c8330ec609a94c59a04fc4bf82c4fced795 Jul 13 22:00:45.768567 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:45.768469 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kr6fz" event={"ID":"9d5db581-bfb8-4fa9-a027-80f98597e1b0","Type":"ContainerStarted","Data":"b27ca5b256a356691f297b87c5107876fa11635044b1795f0102faa3312aaf62"} Jul 13 22:00:45.769567 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:45.769535 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pqjkz" event={"ID":"54f54657-10eb-47ae-ba8f-ec453e686d04","Type":"ContainerStarted","Data":"b29cee968700f47f015ec499de6e6c8330ec609a94c59a04fc4bf82c4fced795"} Jul 13 22:00:50.785894 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:50.785807 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kr6fz" event={"ID":"9d5db581-bfb8-4fa9-a027-80f98597e1b0","Type":"ContainerStarted","Data":"6f2586c0c202b594078707a4da91ed78089c6c37a26cbc170dfe07501a5b362c"} Jul 13 22:00:50.785894 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:50.785863 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kr6fz" event={"ID":"9d5db581-bfb8-4fa9-a027-80f98597e1b0","Type":"ContainerStarted","Data":"5c20867765c5e09ac3d2ced67e05c1b34c41b536fc4d536614ce230e9d357a79"} Jul 13 22:00:50.787478 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:50.787452 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pqjkz" event={"ID":"54f54657-10eb-47ae-ba8f-ec453e686d04","Type":"ContainerStarted","Data":"9249991ce5962ccedeff7599922e17390db857c8ab9041f3027ea71b6e093c7c"} Jul 13 22:00:50.787605 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:50.787492 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pqjkz" event={"ID":"54f54657-10eb-47ae-ba8f-ec453e686d04","Type":"ContainerStarted","Data":"804d276a284430f338ac699896d2753c130df47dd7b26d91d1ccc28c6db13d10"} Jul 13 22:00:50.787666 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:50.787605 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-pqjkz" Jul 13 22:00:50.807342 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:50.807298 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-pqjkz" podStartSLOduration=17.588531689 podStartE2EDuration="21.807281335s" totalImagesPullingTime="4.218749646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:00:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:00:50.804797438 +0000 UTC m=+56.798551751" watchObservedRunningTime="2026-07-13 22:00:50.807281335 +0000 UTC m=+56.801035627" Jul 13 22:00:50.979839 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:50.979808 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7fbd57d5dd-ncm6n"] Jul 13 22:00:50.981709 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:50.981684 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7fbd57d5dd-ncm6n" Jul 13 22:00:50.984663 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:50.984490 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 13 22:00:50.984663 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:50.984519 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 13 22:00:50.984663 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:50.984524 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 13 22:00:50.984663 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:50.984558 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-xpt4p\"" Jul 13 22:00:50.984930 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:50.984664 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 13 22:00:50.986741 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:50.986713 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6477f658fb-sjb79"] Jul 13 22:00:50.988779 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:50.988762 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6477f658fb-sjb79" Jul 13 22:00:50.991731 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:50.991701 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 13 22:00:50.991844 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:50.991746 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 13 22:00:50.991933 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:50.991911 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 13 22:00:50.992030 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:50.991941 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 13 22:00:50.995907 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:50.995885 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7fbd57d5dd-ncm6n"] Jul 13 22:00:51.015871 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:51.013772 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6477f658fb-sjb79"] Jul 13 22:00:51.077552 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:51.077477 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ca272aa2-2fd3-4d0e-b02a-113cf512c66f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6477f658fb-sjb79\" (UID: \"ca272aa2-2fd3-4d0e-b02a-113cf512c66f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6477f658fb-sjb79" Jul 13 22:00:51.077552 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:51.077512 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca272aa2-2fd3-4d0e-b02a-113cf512c66f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6477f658fb-sjb79\" (UID: \"ca272aa2-2fd3-4d0e-b02a-113cf512c66f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6477f658fb-sjb79" Jul 13 22:00:51.077552 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:51.077539 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5sw4\" (UniqueName: \"kubernetes.io/projected/ca272aa2-2fd3-4d0e-b02a-113cf512c66f-kube-api-access-d5sw4\") pod \"cluster-proxy-proxy-agent-6477f658fb-sjb79\" (UID: \"ca272aa2-2fd3-4d0e-b02a-113cf512c66f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6477f658fb-sjb79" Jul 13 22:00:51.077780 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:51.077575 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ca272aa2-2fd3-4d0e-b02a-113cf512c66f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6477f658fb-sjb79\" (UID: \"ca272aa2-2fd3-4d0e-b02a-113cf512c66f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6477f658fb-sjb79" Jul 13 22:00:51.077780 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:51.077615 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ca272aa2-2fd3-4d0e-b02a-113cf512c66f-hub\") pod \"cluster-proxy-proxy-agent-6477f658fb-sjb79\" (UID: \"ca272aa2-2fd3-4d0e-b02a-113cf512c66f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6477f658fb-sjb79" Jul 13 22:00:51.077780 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:51.077633 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/aa02df75-659b-4767-ab43-01d190afcb7e-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7fbd57d5dd-ncm6n\" (UID: \"aa02df75-659b-4767-ab43-01d190afcb7e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7fbd57d5dd-ncm6n" Jul 13 22:00:51.077780 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:51.077649 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ca272aa2-2fd3-4d0e-b02a-113cf512c66f-ca\") pod \"cluster-proxy-proxy-agent-6477f658fb-sjb79\" (UID: \"ca272aa2-2fd3-4d0e-b02a-113cf512c66f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6477f658fb-sjb79" Jul 13 22:00:51.077780 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:51.077674 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlvxd\" (UniqueName: \"kubernetes.io/projected/aa02df75-659b-4767-ab43-01d190afcb7e-kube-api-access-nlvxd\") pod \"managed-serviceaccount-addon-agent-7fbd57d5dd-ncm6n\" (UID: \"aa02df75-659b-4767-ab43-01d190afcb7e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7fbd57d5dd-ncm6n" Jul 13 22:00:51.178773 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:51.178731 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ca272aa2-2fd3-4d0e-b02a-113cf512c66f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6477f658fb-sjb79\" (UID: \"ca272aa2-2fd3-4d0e-b02a-113cf512c66f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6477f658fb-sjb79" Jul 13 22:00:51.178948 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:51.178782 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca272aa2-2fd3-4d0e-b02a-113cf512c66f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6477f658fb-sjb79\" (UID: \"ca272aa2-2fd3-4d0e-b02a-113cf512c66f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6477f658fb-sjb79" Jul 13 22:00:51.178948 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:51.178814 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5sw4\" (UniqueName: \"kubernetes.io/projected/ca272aa2-2fd3-4d0e-b02a-113cf512c66f-kube-api-access-d5sw4\") pod \"cluster-proxy-proxy-agent-6477f658fb-sjb79\" (UID: \"ca272aa2-2fd3-4d0e-b02a-113cf512c66f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6477f658fb-sjb79" Jul 13 22:00:51.178948 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:51.178842 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ca272aa2-2fd3-4d0e-b02a-113cf512c66f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6477f658fb-sjb79\" (UID: \"ca272aa2-2fd3-4d0e-b02a-113cf512c66f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6477f658fb-sjb79" Jul 13 22:00:51.178948 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:51.178869 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ca272aa2-2fd3-4d0e-b02a-113cf512c66f-hub\") pod \"cluster-proxy-proxy-agent-6477f658fb-sjb79\" (UID: \"ca272aa2-2fd3-4d0e-b02a-113cf512c66f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6477f658fb-sjb79" Jul 13 22:00:51.178948 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:51.178896 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/aa02df75-659b-4767-ab43-01d190afcb7e-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7fbd57d5dd-ncm6n\" (UID: \"aa02df75-659b-4767-ab43-01d190afcb7e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7fbd57d5dd-ncm6n" Jul 13 22:00:51.178948 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:51.178921 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ca272aa2-2fd3-4d0e-b02a-113cf512c66f-ca\") pod \"cluster-proxy-proxy-agent-6477f658fb-sjb79\" (UID: \"ca272aa2-2fd3-4d0e-b02a-113cf512c66f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6477f658fb-sjb79" Jul 13 22:00:51.178948 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:51.178949 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlvxd\" (UniqueName: \"kubernetes.io/projected/aa02df75-659b-4767-ab43-01d190afcb7e-kube-api-access-nlvxd\") pod \"managed-serviceaccount-addon-agent-7fbd57d5dd-ncm6n\" (UID: \"aa02df75-659b-4767-ab43-01d190afcb7e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7fbd57d5dd-ncm6n" Jul 13 22:00:51.179999 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:51.179932 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca272aa2-2fd3-4d0e-b02a-113cf512c66f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6477f658fb-sjb79\" (UID: \"ca272aa2-2fd3-4d0e-b02a-113cf512c66f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6477f658fb-sjb79" Jul 13 22:00:51.181906 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:51.181878 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ca272aa2-2fd3-4d0e-b02a-113cf512c66f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6477f658fb-sjb79\" (UID: \"ca272aa2-2fd3-4d0e-b02a-113cf512c66f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6477f658fb-sjb79" Jul 13 22:00:51.182527 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:51.182501 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ca272aa2-2fd3-4d0e-b02a-113cf512c66f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6477f658fb-sjb79\" (UID: \"ca272aa2-2fd3-4d0e-b02a-113cf512c66f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6477f658fb-sjb79" Jul 13 22:00:51.182743 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:51.182718 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ca272aa2-2fd3-4d0e-b02a-113cf512c66f-hub\") pod \"cluster-proxy-proxy-agent-6477f658fb-sjb79\" (UID: \"ca272aa2-2fd3-4d0e-b02a-113cf512c66f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6477f658fb-sjb79" Jul 13 22:00:51.182813 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:51.182779 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/aa02df75-659b-4767-ab43-01d190afcb7e-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7fbd57d5dd-ncm6n\" (UID: \"aa02df75-659b-4767-ab43-01d190afcb7e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7fbd57d5dd-ncm6n" Jul 13 22:00:51.182878 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:51.182721 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ca272aa2-2fd3-4d0e-b02a-113cf512c66f-ca\") pod \"cluster-proxy-proxy-agent-6477f658fb-sjb79\" (UID: \"ca272aa2-2fd3-4d0e-b02a-113cf512c66f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6477f658fb-sjb79" Jul 13 22:00:51.189335 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:51.189302 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlvxd\" (UniqueName: \"kubernetes.io/projected/aa02df75-659b-4767-ab43-01d190afcb7e-kube-api-access-nlvxd\") pod \"managed-serviceaccount-addon-agent-7fbd57d5dd-ncm6n\" (UID: \"aa02df75-659b-4767-ab43-01d190afcb7e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7fbd57d5dd-ncm6n" Jul 13 22:00:51.191038 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:51.191012 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5sw4\" (UniqueName: \"kubernetes.io/projected/ca272aa2-2fd3-4d0e-b02a-113cf512c66f-kube-api-access-d5sw4\") pod \"cluster-proxy-proxy-agent-6477f658fb-sjb79\" (UID: \"ca272aa2-2fd3-4d0e-b02a-113cf512c66f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6477f658fb-sjb79" Jul 13 22:00:51.301971 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:51.301932 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7fbd57d5dd-ncm6n" Jul 13 22:00:51.308602 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:51.308580 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6477f658fb-sjb79" Jul 13 22:00:51.451612 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:51.451575 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7fbd57d5dd-ncm6n"] Jul 13 22:00:51.455460 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:00:51.455419 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa02df75_659b_4767_ab43_01d190afcb7e.slice/crio-6956481c31423e7a514b2fc6a7e7f8eb2677a5e9b48f09e8452e9ca2cb874d2a WatchSource:0}: Error finding container 6956481c31423e7a514b2fc6a7e7f8eb2677a5e9b48f09e8452e9ca2cb874d2a: Status 404 returned error can't find the container with id 6956481c31423e7a514b2fc6a7e7f8eb2677a5e9b48f09e8452e9ca2cb874d2a Jul 13 22:00:51.476687 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:51.476653 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6477f658fb-sjb79"] Jul 13 22:00:51.480151 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:00:51.480123 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca272aa2_2fd3_4d0e_b02a_113cf512c66f.slice/crio-b8931b4e4a1bfa795023cb80223a2c7b53f8ff72589f00dd1bb9ad27f720bcf2 WatchSource:0}: Error finding container b8931b4e4a1bfa795023cb80223a2c7b53f8ff72589f00dd1bb9ad27f720bcf2: Status 404 returned error can't find the container with id b8931b4e4a1bfa795023cb80223a2c7b53f8ff72589f00dd1bb9ad27f720bcf2 Jul 13 22:00:51.792989 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:51.792947 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7fbd57d5dd-ncm6n" event={"ID":"aa02df75-659b-4767-ab43-01d190afcb7e","Type":"ContainerStarted","Data":"6956481c31423e7a514b2fc6a7e7f8eb2677a5e9b48f09e8452e9ca2cb874d2a"} Jul 13 22:00:51.794082 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:51.794056 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6477f658fb-sjb79" event={"ID":"ca272aa2-2fd3-4d0e-b02a-113cf512c66f","Type":"ContainerStarted","Data":"b8931b4e4a1bfa795023cb80223a2c7b53f8ff72589f00dd1bb9ad27f720bcf2"} Jul 13 22:00:53.715977 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:53.715687 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tbknb" Jul 13 22:00:53.806830 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:53.806783 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kr6fz" event={"ID":"9d5db581-bfb8-4fa9-a027-80f98597e1b0","Type":"ContainerStarted","Data":"4f24cde8f61ae547744e49ec6880f6fbecbb687ad7cc4b6b51e8d8d8247a7729"} Jul 13 22:00:56.818676 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:56.818582 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6477f658fb-sjb79" event={"ID":"ca272aa2-2fd3-4d0e-b02a-113cf512c66f","Type":"ContainerStarted","Data":"967ffa9d8b678ebf8516589cf858a5d8a9ee1e7fc8c0764023d86f9c2c3f4c88"} Jul 13 22:00:56.818676 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:56.818636 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6477f658fb-sjb79" event={"ID":"ca272aa2-2fd3-4d0e-b02a-113cf512c66f","Type":"ContainerStarted","Data":"660ce63ea5fe7ec6bba3ce1e9a47e3e7fba1774a6534fe8c4b63fbf2bf4a0187"} Jul 13 22:00:56.818676 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:56.818649 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6477f658fb-sjb79" event={"ID":"ca272aa2-2fd3-4d0e-b02a-113cf512c66f","Type":"ContainerStarted","Data":"73b32cbda2423023025f067c8d5fe2a71a6768f3800987a6d31f604a414cbfd6"} Jul 13 22:00:56.820060 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:56.820027 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7fbd57d5dd-ncm6n" event={"ID":"aa02df75-659b-4767-ab43-01d190afcb7e","Type":"ContainerStarted","Data":"6c1d4a9ba93722ddbff0ffd17ad8c40e3bb5a42af1963c77a57bee12fe8ca713"} Jul 13 22:00:56.841962 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:56.839552 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-kr6fz" podStartSLOduration=20.496587807 podStartE2EDuration="27.839532384s" totalImagesPullingTime="7.342944577s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 22:00:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:00:53.83137147 +0000 UTC m=+59.825125763" watchObservedRunningTime="2026-07-13 22:00:56.839532384 +0000 UTC m=+62.833286677" Jul 13 22:00:56.841962 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:56.840022 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6477f658fb-sjb79" podStartSLOduration=1.788742394 podStartE2EDuration="6.840007477s" totalImagesPullingTime="5.051265083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:00:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:00:56.838157006 +0000 UTC m=+62.831911300" watchObservedRunningTime="2026-07-13 22:00:56.840007477 +0000 UTC m=+62.833761770" Jul 13 22:00:56.858800 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:00:56.858759 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7fbd57d5dd-ncm6n" podStartSLOduration=1.795101641 podStartE2EDuration="6.85874502s" totalImagesPullingTime="5.063643379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:00:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:00:56.857722237 +0000 UTC m=+62.851476524" watchObservedRunningTime="2026-07-13 22:00:56.85874502 +0000 UTC m=+62.852499335" Jul 13 22:01:00.354766 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:00.354723 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56086743-e831-415a-a2c3-6982ef099751-original-pull-secret\") pod \"global-pull-secret-syncer-5bkm9\" (UID: \"56086743-e831-415a-a2c3-6982ef099751\") " pod="kube-system/global-pull-secret-syncer-5bkm9" Jul 13 22:01:00.355239 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:00.354782 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55ac7566-430e-4e45-ab69-fe943cfb4c44-metrics-certs\") pod \"network-metrics-daemon-87w2d\" (UID: \"55ac7566-430e-4e45-ab69-fe943cfb4c44\") " pod="openshift-multus/network-metrics-daemon-87w2d" Jul 13 22:01:00.358475 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:00.358457 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 13 22:01:00.358727 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:00.358714 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 13 22:01:00.368237 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:00.368194 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56086743-e831-415a-a2c3-6982ef099751-original-pull-secret\") pod \"global-pull-secret-syncer-5bkm9\" (UID: \"56086743-e831-415a-a2c3-6982ef099751\") " pod="kube-system/global-pull-secret-syncer-5bkm9" Jul 13 22:01:00.368380 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:00.368234 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55ac7566-430e-4e45-ab69-fe943cfb4c44-metrics-certs\") pod \"network-metrics-daemon-87w2d\" (UID: \"55ac7566-430e-4e45-ab69-fe943cfb4c44\") " pod="openshift-multus/network-metrics-daemon-87w2d" Jul 13 22:01:00.455682 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:00.455640 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8h4k\" (UniqueName: \"kubernetes.io/projected/d1b9ac92-b63a-4180-9293-86c6b0c545a0-kube-api-access-h8h4k\") pod \"network-check-target-rg8tn\" (UID: \"d1b9ac92-b63a-4180-9293-86c6b0c545a0\") " pod="openshift-network-diagnostics/network-check-target-rg8tn" Jul 13 22:01:00.458085 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:00.458066 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 13 22:01:00.468778 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:00.468756 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 13 22:01:00.473460 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:00.473442 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-b6tsw\"" Jul 13 22:01:00.476581 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:00.476564 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5bkm9" Jul 13 22:01:00.479142 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:00.479122 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8h4k\" (UniqueName: \"kubernetes.io/projected/d1b9ac92-b63a-4180-9293-86c6b0c545a0-kube-api-access-h8h4k\") pod \"network-check-target-rg8tn\" (UID: \"d1b9ac92-b63a-4180-9293-86c6b0c545a0\") " pod="openshift-network-diagnostics/network-check-target-rg8tn" Jul 13 22:01:00.481451 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:00.481433 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87w2d" Jul 13 22:01:00.615339 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:00.615239 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5bkm9"] Jul 13 22:01:00.618624 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:01:00.618571 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56086743_e831_415a_a2c3_6982ef099751.slice/crio-715f544ac55b083fa794b73b9610213660c5627751093960f9025db44555307a WatchSource:0}: Error finding container 715f544ac55b083fa794b73b9610213660c5627751093960f9025db44555307a: Status 404 returned error can't find the container with id 715f544ac55b083fa794b73b9610213660c5627751093960f9025db44555307a Jul 13 22:01:00.632866 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:00.632844 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-87w2d"] Jul 13 22:01:00.635589 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:01:00.635568 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55ac7566_430e_4e45_ab69_fe943cfb4c44.slice/crio-88ff7912e0c411d2f016b2bdc82ea0ef236c70bd65078434a11129ef3947a9f8 WatchSource:0}: Error finding container 88ff7912e0c411d2f016b2bdc82ea0ef236c70bd65078434a11129ef3947a9f8: Status 404 returned error can't find the container with id 88ff7912e0c411d2f016b2bdc82ea0ef236c70bd65078434a11129ef3947a9f8 Jul 13 22:01:00.768057 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:00.768024 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gmvbg\"" Jul 13 22:01:00.776227 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:00.776178 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rg8tn" Jul 13 22:01:00.796477 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:00.796453 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pqjkz" Jul 13 22:01:00.834404 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:00.834323 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5bkm9" event={"ID":"56086743-e831-415a-a2c3-6982ef099751","Type":"ContainerStarted","Data":"715f544ac55b083fa794b73b9610213660c5627751093960f9025db44555307a"} Jul 13 22:01:00.836131 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:00.836031 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-87w2d" event={"ID":"55ac7566-430e-4e45-ab69-fe943cfb4c44","Type":"ContainerStarted","Data":"88ff7912e0c411d2f016b2bdc82ea0ef236c70bd65078434a11129ef3947a9f8"} Jul 13 22:01:00.921187 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:00.921159 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rg8tn"] Jul 13 22:01:00.924751 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:01:00.924722 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1b9ac92_b63a_4180_9293_86c6b0c545a0.slice/crio-294f24c826814a2f52bb37a71badde1a1709806fc93fdaef17cdcccfcc459ca2 WatchSource:0}: Error finding container 294f24c826814a2f52bb37a71badde1a1709806fc93fdaef17cdcccfcc459ca2: Status 404 returned error can't find the container with id 294f24c826814a2f52bb37a71badde1a1709806fc93fdaef17cdcccfcc459ca2 Jul 13 22:01:01.842146 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:01.842068 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rg8tn" event={"ID":"d1b9ac92-b63a-4180-9293-86c6b0c545a0","Type":"ContainerStarted","Data":"294f24c826814a2f52bb37a71badde1a1709806fc93fdaef17cdcccfcc459ca2"} Jul 13 22:01:02.848893 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:02.848784 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-87w2d" event={"ID":"55ac7566-430e-4e45-ab69-fe943cfb4c44","Type":"ContainerStarted","Data":"a3968895c46019190cbbdd796bc823b2022787e0245f5e23973e1857f7de7594"} Jul 13 22:01:02.848893 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:02.848851 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-87w2d" event={"ID":"55ac7566-430e-4e45-ab69-fe943cfb4c44","Type":"ContainerStarted","Data":"54675ccac86413bce68b5401b554af1e6a3a55d31f4a4e773a296dce2dd384be"} Jul 13 22:01:02.865096 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:02.865018 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-87w2d" podStartSLOduration=67.106709326 podStartE2EDuration="1m8.864996635s" totalImagesPullingTime="1.758287309s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:59:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:01:02.863528588 +0000 UTC m=+68.857282883" watchObservedRunningTime="2026-07-13 22:01:02.864996635 +0000 UTC m=+68.858750929" Jul 13 22:01:05.862322 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:05.862220 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5bkm9" event={"ID":"56086743-e831-415a-a2c3-6982ef099751","Type":"ContainerStarted","Data":"2d7fa1e2e3b89f131db8a037c6b0028bc73cab02f4c78a7c6f277467ce8d0aee"} Jul 13 22:01:05.863782 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:05.863757 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rg8tn" event={"ID":"d1b9ac92-b63a-4180-9293-86c6b0c545a0","Type":"ContainerStarted","Data":"bea90b5f52c926895d50df879caf956bc1a119e0381badb0bbfc85293dc1a678"} Jul 13 22:01:05.863915 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:05.863897 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-rg8tn" Jul 13 22:01:05.879884 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:05.879832 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-5bkm9" podStartSLOduration=67.012639821 podStartE2EDuration="1m11.879813026s" totalImagesPullingTime="4.867173205s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 21:59:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:01:05.878927885 +0000 UTC m=+71.872682174" watchObservedRunningTime="2026-07-13 22:01:05.879813026 +0000 UTC m=+71.873567320" Jul 13 22:01:05.895872 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:05.895835 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-rg8tn" podStartSLOduration=67.33760451 podStartE2EDuration="1m11.895821637s" totalImagesPullingTime="4.558217127s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:59:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:01:05.895490141 +0000 UTC m=+71.889244461" watchObservedRunningTime="2026-07-13 22:01:05.895821637 +0000 UTC m=+71.889575932" Jul 13 22:01:06.082373 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:06.082339 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-5klkq"] Jul 13 22:01:06.084511 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:06.084489 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5klkq" Jul 13 22:01:06.087419 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:06.087393 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 13 22:01:06.087726 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:06.087702 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 13 22:01:06.087985 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:06.087969 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 13 22:01:06.088523 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:06.088506 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 13 22:01:06.088621 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:06.088569 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 13 22:01:06.088621 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:06.088617 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 13 22:01:06.088732 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:06.088574 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-k57rd\"" Jul 13 22:01:06.203801 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:06.203767 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3982d16e-b33e-4fc4-b834-bd8f9b6a35ce-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5klkq\" (UID: \"3982d16e-b33e-4fc4-b834-bd8f9b6a35ce\") " pod="openshift-monitoring/node-exporter-5klkq" Jul 13 22:01:06.203997 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:06.203815 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3982d16e-b33e-4fc4-b834-bd8f9b6a35ce-node-exporter-accelerators-collector-config\") pod \"node-exporter-5klkq\" (UID: \"3982d16e-b33e-4fc4-b834-bd8f9b6a35ce\") " pod="openshift-monitoring/node-exporter-5klkq" Jul 13 22:01:06.203997 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:06.203841 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3982d16e-b33e-4fc4-b834-bd8f9b6a35ce-node-exporter-textfile\") pod \"node-exporter-5klkq\" (UID: \"3982d16e-b33e-4fc4-b834-bd8f9b6a35ce\") " pod="openshift-monitoring/node-exporter-5klkq" Jul 13 22:01:06.203997 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:06.203867 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t54kp\" (UniqueName: \"kubernetes.io/projected/3982d16e-b33e-4fc4-b834-bd8f9b6a35ce-kube-api-access-t54kp\") pod \"node-exporter-5klkq\" (UID: \"3982d16e-b33e-4fc4-b834-bd8f9b6a35ce\") " pod="openshift-monitoring/node-exporter-5klkq" Jul 13 22:01:06.203997 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:06.203890 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3982d16e-b33e-4fc4-b834-bd8f9b6a35ce-node-exporter-wtmp\") pod \"node-exporter-5klkq\" (UID: \"3982d16e-b33e-4fc4-b834-bd8f9b6a35ce\") " pod="openshift-monitoring/node-exporter-5klkq" Jul 13 22:01:06.203997 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:06.203915 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3982d16e-b33e-4fc4-b834-bd8f9b6a35ce-sys\") pod \"node-exporter-5klkq\" (UID: \"3982d16e-b33e-4fc4-b834-bd8f9b6a35ce\") " pod="openshift-monitoring/node-exporter-5klkq" Jul 13 22:01:06.203997 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:06.203947 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3982d16e-b33e-4fc4-b834-bd8f9b6a35ce-node-exporter-tls\") pod \"node-exporter-5klkq\" (UID: \"3982d16e-b33e-4fc4-b834-bd8f9b6a35ce\") " pod="openshift-monitoring/node-exporter-5klkq" Jul 13 22:01:06.203997 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:06.203966 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3982d16e-b33e-4fc4-b834-bd8f9b6a35ce-metrics-client-ca\") pod \"node-exporter-5klkq\" (UID: \"3982d16e-b33e-4fc4-b834-bd8f9b6a35ce\") " pod="openshift-monitoring/node-exporter-5klkq" Jul 13 22:01:06.203997 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:06.203983 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3982d16e-b33e-4fc4-b834-bd8f9b6a35ce-root\") pod \"node-exporter-5klkq\" (UID: \"3982d16e-b33e-4fc4-b834-bd8f9b6a35ce\") " pod="openshift-monitoring/node-exporter-5klkq" Jul 13 22:01:06.305250 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:06.305215 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3982d16e-b33e-4fc4-b834-bd8f9b6a35ce-node-exporter-accelerators-collector-config\") pod \"node-exporter-5klkq\" (UID: \"3982d16e-b33e-4fc4-b834-bd8f9b6a35ce\") " pod="openshift-monitoring/node-exporter-5klkq" Jul 13 22:01:06.305464 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:06.305261 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3982d16e-b33e-4fc4-b834-bd8f9b6a35ce-node-exporter-textfile\") pod \"node-exporter-5klkq\" (UID: \"3982d16e-b33e-4fc4-b834-bd8f9b6a35ce\") " pod="openshift-monitoring/node-exporter-5klkq" Jul 13 22:01:06.305464 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:06.305290 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t54kp\" (UniqueName: \"kubernetes.io/projected/3982d16e-b33e-4fc4-b834-bd8f9b6a35ce-kube-api-access-t54kp\") pod \"node-exporter-5klkq\" (UID: \"3982d16e-b33e-4fc4-b834-bd8f9b6a35ce\") " pod="openshift-monitoring/node-exporter-5klkq" Jul 13 22:01:06.305464 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:06.305322 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3982d16e-b33e-4fc4-b834-bd8f9b6a35ce-node-exporter-wtmp\") pod \"node-exporter-5klkq\" (UID: \"3982d16e-b33e-4fc4-b834-bd8f9b6a35ce\") " pod="openshift-monitoring/node-exporter-5klkq" Jul 13 22:01:06.305464 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:06.305348 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3982d16e-b33e-4fc4-b834-bd8f9b6a35ce-sys\") pod \"node-exporter-5klkq\" (UID: \"3982d16e-b33e-4fc4-b834-bd8f9b6a35ce\") " pod="openshift-monitoring/node-exporter-5klkq" Jul 13 22:01:06.305678 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:06.305480 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3982d16e-b33e-4fc4-b834-bd8f9b6a35ce-node-exporter-wtmp\") pod \"node-exporter-5klkq\" (UID: \"3982d16e-b33e-4fc4-b834-bd8f9b6a35ce\") " pod="openshift-monitoring/node-exporter-5klkq" Jul 13 22:01:06.305678 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:06.305514 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3982d16e-b33e-4fc4-b834-bd8f9b6a35ce-node-exporter-tls\") pod \"node-exporter-5klkq\" (UID: \"3982d16e-b33e-4fc4-b834-bd8f9b6a35ce\") " pod="openshift-monitoring/node-exporter-5klkq" Jul 13 22:01:06.305678 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:06.305548 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3982d16e-b33e-4fc4-b834-bd8f9b6a35ce-metrics-client-ca\") pod \"node-exporter-5klkq\" (UID: \"3982d16e-b33e-4fc4-b834-bd8f9b6a35ce\") " pod="openshift-monitoring/node-exporter-5klkq" Jul 13 22:01:06.305678 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:06.305578 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3982d16e-b33e-4fc4-b834-bd8f9b6a35ce-root\") pod \"node-exporter-5klkq\" (UID: \"3982d16e-b33e-4fc4-b834-bd8f9b6a35ce\") " pod="openshift-monitoring/node-exporter-5klkq" Jul 13 22:01:06.305678 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:06.305610 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3982d16e-b33e-4fc4-b834-bd8f9b6a35ce-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5klkq\" (UID: \"3982d16e-b33e-4fc4-b834-bd8f9b6a35ce\") " pod="openshift-monitoring/node-exporter-5klkq" Jul 13 22:01:06.305910 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:06.305685 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3982d16e-b33e-4fc4-b834-bd8f9b6a35ce-sys\") pod \"node-exporter-5klkq\" (UID: \"3982d16e-b33e-4fc4-b834-bd8f9b6a35ce\") " pod="openshift-monitoring/node-exporter-5klkq" Jul 13 22:01:06.305910 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:06.305703 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3982d16e-b33e-4fc4-b834-bd8f9b6a35ce-node-exporter-textfile\") pod \"node-exporter-5klkq\" (UID: \"3982d16e-b33e-4fc4-b834-bd8f9b6a35ce\") " pod="openshift-monitoring/node-exporter-5klkq" Jul 13 22:01:06.305910 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:06.305770 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3982d16e-b33e-4fc4-b834-bd8f9b6a35ce-root\") pod \"node-exporter-5klkq\" (UID: \"3982d16e-b33e-4fc4-b834-bd8f9b6a35ce\") " pod="openshift-monitoring/node-exporter-5klkq" Jul 13 22:01:06.305910 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:01:06.305848 2555 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 13 22:01:06.305910 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:01:06.305903 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3982d16e-b33e-4fc4-b834-bd8f9b6a35ce-node-exporter-tls podName:3982d16e-b33e-4fc4-b834-bd8f9b6a35ce nodeName:}" failed. No retries permitted until 2026-07-13 22:01:06.805885272 +0000 UTC m=+72.799639572 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/3982d16e-b33e-4fc4-b834-bd8f9b6a35ce-node-exporter-tls") pod "node-exporter-5klkq" (UID: "3982d16e-b33e-4fc4-b834-bd8f9b6a35ce") : secret "node-exporter-tls" not found Jul 13 22:01:06.306230 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:06.306020 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3982d16e-b33e-4fc4-b834-bd8f9b6a35ce-node-exporter-accelerators-collector-config\") pod \"node-exporter-5klkq\" (UID: \"3982d16e-b33e-4fc4-b834-bd8f9b6a35ce\") " pod="openshift-monitoring/node-exporter-5klkq" Jul 13 22:01:06.306230 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:06.306168 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3982d16e-b33e-4fc4-b834-bd8f9b6a35ce-metrics-client-ca\") pod \"node-exporter-5klkq\" (UID: \"3982d16e-b33e-4fc4-b834-bd8f9b6a35ce\") " pod="openshift-monitoring/node-exporter-5klkq" Jul 13 22:01:06.308048 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:06.308016 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3982d16e-b33e-4fc4-b834-bd8f9b6a35ce-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5klkq\" (UID: \"3982d16e-b33e-4fc4-b834-bd8f9b6a35ce\") " pod="openshift-monitoring/node-exporter-5klkq" Jul 13 22:01:06.320687 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:06.320658 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t54kp\" (UniqueName: \"kubernetes.io/projected/3982d16e-b33e-4fc4-b834-bd8f9b6a35ce-kube-api-access-t54kp\") pod \"node-exporter-5klkq\" (UID: \"3982d16e-b33e-4fc4-b834-bd8f9b6a35ce\") " pod="openshift-monitoring/node-exporter-5klkq" Jul 13 22:01:06.809455 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:06.809408 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3982d16e-b33e-4fc4-b834-bd8f9b6a35ce-node-exporter-tls\") pod \"node-exporter-5klkq\" (UID: \"3982d16e-b33e-4fc4-b834-bd8f9b6a35ce\") " pod="openshift-monitoring/node-exporter-5klkq" Jul 13 22:01:06.811871 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:06.811847 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3982d16e-b33e-4fc4-b834-bd8f9b6a35ce-node-exporter-tls\") pod \"node-exporter-5klkq\" (UID: \"3982d16e-b33e-4fc4-b834-bd8f9b6a35ce\") " pod="openshift-monitoring/node-exporter-5klkq" Jul 13 22:01:06.993928 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:06.993888 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5klkq" Jul 13 22:01:07.005135 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:01:07.005105 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3982d16e_b33e_4fc4_b834_bd8f9b6a35ce.slice/crio-5068d0d7b7ff1f6877f056e0dceef4507d078eac2135213c9f594e19d56f45c4 WatchSource:0}: Error finding container 5068d0d7b7ff1f6877f056e0dceef4507d078eac2135213c9f594e19d56f45c4: Status 404 returned error can't find the container with id 5068d0d7b7ff1f6877f056e0dceef4507d078eac2135213c9f594e19d56f45c4 Jul 13 22:01:07.873698 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:07.873652 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5klkq" event={"ID":"3982d16e-b33e-4fc4-b834-bd8f9b6a35ce","Type":"ContainerStarted","Data":"5068d0d7b7ff1f6877f056e0dceef4507d078eac2135213c9f594e19d56f45c4"} Jul 13 22:01:08.879580 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:08.879549 2555 generic.go:356] "Generic (PLEG): container finished" podID="3982d16e-b33e-4fc4-b834-bd8f9b6a35ce" containerID="2dcda883eee39ea4a23a1af4069081780206e27fafc718d896ebfa9f7304b0d4" exitCode=0 Jul 13 22:01:08.879955 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:08.879616 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5klkq" event={"ID":"3982d16e-b33e-4fc4-b834-bd8f9b6a35ce","Type":"ContainerDied","Data":"2dcda883eee39ea4a23a1af4069081780206e27fafc718d896ebfa9f7304b0d4"} Jul 13 22:01:09.305512 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:09.305430 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-b49b6899d-nxgfv"] Jul 13 22:01:09.309311 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:09.309291 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-b49b6899d-nxgfv" Jul 13 22:01:09.314989 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:09.314961 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-xbln2\"" Jul 13 22:01:09.314989 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:09.314985 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 13 22:01:09.315154 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:09.314998 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 13 22:01:09.315154 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:09.315023 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 13 22:01:09.315154 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:09.315083 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 13 22:01:09.315314 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:09.315223 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 13 22:01:09.315466 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:09.315445 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-bc422qjlkrnv6\"" Jul 13 22:01:09.329226 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:09.329187 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-b49b6899d-nxgfv"] Jul 13 22:01:09.426435 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:09.426394 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f4ff6c51-71c0-4f77-a4ef-ef5b602360fc-secret-thanos-querier-tls\") pod \"thanos-querier-b49b6899d-nxgfv\" (UID: \"f4ff6c51-71c0-4f77-a4ef-ef5b602360fc\") " pod="openshift-monitoring/thanos-querier-b49b6899d-nxgfv" Jul 13 22:01:09.426601 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:09.426440 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f4ff6c51-71c0-4f77-a4ef-ef5b602360fc-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-b49b6899d-nxgfv\" (UID: \"f4ff6c51-71c0-4f77-a4ef-ef5b602360fc\") " pod="openshift-monitoring/thanos-querier-b49b6899d-nxgfv" Jul 13 22:01:09.426601 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:09.426462 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f4ff6c51-71c0-4f77-a4ef-ef5b602360fc-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-b49b6899d-nxgfv\" (UID: \"f4ff6c51-71c0-4f77-a4ef-ef5b602360fc\") " pod="openshift-monitoring/thanos-querier-b49b6899d-nxgfv" Jul 13 22:01:09.426601 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:09.426541 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftvs5\" (UniqueName: \"kubernetes.io/projected/f4ff6c51-71c0-4f77-a4ef-ef5b602360fc-kube-api-access-ftvs5\") pod \"thanos-querier-b49b6899d-nxgfv\" (UID: \"f4ff6c51-71c0-4f77-a4ef-ef5b602360fc\") " pod="openshift-monitoring/thanos-querier-b49b6899d-nxgfv" Jul 13 22:01:09.426601 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:09.426589 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f4ff6c51-71c0-4f77-a4ef-ef5b602360fc-secret-grpc-tls\") pod \"thanos-querier-b49b6899d-nxgfv\" (UID: \"f4ff6c51-71c0-4f77-a4ef-ef5b602360fc\") " pod="openshift-monitoring/thanos-querier-b49b6899d-nxgfv" Jul 13 22:01:09.426739 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:09.426614 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f4ff6c51-71c0-4f77-a4ef-ef5b602360fc-metrics-client-ca\") pod \"thanos-querier-b49b6899d-nxgfv\" (UID: \"f4ff6c51-71c0-4f77-a4ef-ef5b602360fc\") " pod="openshift-monitoring/thanos-querier-b49b6899d-nxgfv" Jul 13 22:01:09.426739 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:09.426644 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f4ff6c51-71c0-4f77-a4ef-ef5b602360fc-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-b49b6899d-nxgfv\" (UID: \"f4ff6c51-71c0-4f77-a4ef-ef5b602360fc\") " pod="openshift-monitoring/thanos-querier-b49b6899d-nxgfv" Jul 13 22:01:09.426739 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:09.426693 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f4ff6c51-71c0-4f77-a4ef-ef5b602360fc-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-b49b6899d-nxgfv\" (UID: \"f4ff6c51-71c0-4f77-a4ef-ef5b602360fc\") " pod="openshift-monitoring/thanos-querier-b49b6899d-nxgfv" Jul 13 22:01:09.527629 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:09.527589 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f4ff6c51-71c0-4f77-a4ef-ef5b602360fc-secret-thanos-querier-tls\") pod \"thanos-querier-b49b6899d-nxgfv\" (UID: \"f4ff6c51-71c0-4f77-a4ef-ef5b602360fc\") " pod="openshift-monitoring/thanos-querier-b49b6899d-nxgfv" Jul 13 22:01:09.527795 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:09.527633 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f4ff6c51-71c0-4f77-a4ef-ef5b602360fc-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-b49b6899d-nxgfv\" (UID: \"f4ff6c51-71c0-4f77-a4ef-ef5b602360fc\") " pod="openshift-monitoring/thanos-querier-b49b6899d-nxgfv" Jul 13 22:01:09.527795 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:09.527671 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f4ff6c51-71c0-4f77-a4ef-ef5b602360fc-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-b49b6899d-nxgfv\" (UID: \"f4ff6c51-71c0-4f77-a4ef-ef5b602360fc\") " pod="openshift-monitoring/thanos-querier-b49b6899d-nxgfv" Jul 13 22:01:09.527795 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:09.527706 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftvs5\" (UniqueName: \"kubernetes.io/projected/f4ff6c51-71c0-4f77-a4ef-ef5b602360fc-kube-api-access-ftvs5\") pod \"thanos-querier-b49b6899d-nxgfv\" (UID: \"f4ff6c51-71c0-4f77-a4ef-ef5b602360fc\") " pod="openshift-monitoring/thanos-querier-b49b6899d-nxgfv" Jul 13 22:01:09.527795 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:09.527737 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f4ff6c51-71c0-4f77-a4ef-ef5b602360fc-secret-grpc-tls\") pod \"thanos-querier-b49b6899d-nxgfv\" (UID: \"f4ff6c51-71c0-4f77-a4ef-ef5b602360fc\") " pod="openshift-monitoring/thanos-querier-b49b6899d-nxgfv" Jul 13 22:01:09.527795 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:09.527766 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f4ff6c51-71c0-4f77-a4ef-ef5b602360fc-metrics-client-ca\") pod \"thanos-querier-b49b6899d-nxgfv\" (UID: \"f4ff6c51-71c0-4f77-a4ef-ef5b602360fc\") " pod="openshift-monitoring/thanos-querier-b49b6899d-nxgfv" Jul 13 22:01:09.528055 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:09.527808 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f4ff6c51-71c0-4f77-a4ef-ef5b602360fc-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-b49b6899d-nxgfv\" (UID: \"f4ff6c51-71c0-4f77-a4ef-ef5b602360fc\") " pod="openshift-monitoring/thanos-querier-b49b6899d-nxgfv" Jul 13 22:01:09.528055 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:09.527843 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f4ff6c51-71c0-4f77-a4ef-ef5b602360fc-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-b49b6899d-nxgfv\" (UID: \"f4ff6c51-71c0-4f77-a4ef-ef5b602360fc\") " pod="openshift-monitoring/thanos-querier-b49b6899d-nxgfv" Jul 13 22:01:09.528655 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:09.528631 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f4ff6c51-71c0-4f77-a4ef-ef5b602360fc-metrics-client-ca\") pod \"thanos-querier-b49b6899d-nxgfv\" (UID: \"f4ff6c51-71c0-4f77-a4ef-ef5b602360fc\") " pod="openshift-monitoring/thanos-querier-b49b6899d-nxgfv" Jul 13 22:01:09.530571 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:09.530552 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f4ff6c51-71c0-4f77-a4ef-ef5b602360fc-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-b49b6899d-nxgfv\" (UID: \"f4ff6c51-71c0-4f77-a4ef-ef5b602360fc\") " pod="openshift-monitoring/thanos-querier-b49b6899d-nxgfv" Jul 13 22:01:09.530571 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:09.530562 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f4ff6c51-71c0-4f77-a4ef-ef5b602360fc-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-b49b6899d-nxgfv\" (UID: \"f4ff6c51-71c0-4f77-a4ef-ef5b602360fc\") " pod="openshift-monitoring/thanos-querier-b49b6899d-nxgfv" Jul 13 22:01:09.530869 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:09.530839 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f4ff6c51-71c0-4f77-a4ef-ef5b602360fc-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-b49b6899d-nxgfv\" (UID: \"f4ff6c51-71c0-4f77-a4ef-ef5b602360fc\") " pod="openshift-monitoring/thanos-querier-b49b6899d-nxgfv" Jul 13 22:01:09.530951 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:09.530934 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f4ff6c51-71c0-4f77-a4ef-ef5b602360fc-secret-thanos-querier-tls\") pod \"thanos-querier-b49b6899d-nxgfv\" (UID: \"f4ff6c51-71c0-4f77-a4ef-ef5b602360fc\") " pod="openshift-monitoring/thanos-querier-b49b6899d-nxgfv" Jul 13 22:01:09.531046 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:09.531030 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f4ff6c51-71c0-4f77-a4ef-ef5b602360fc-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-b49b6899d-nxgfv\" (UID: \"f4ff6c51-71c0-4f77-a4ef-ef5b602360fc\") " pod="openshift-monitoring/thanos-querier-b49b6899d-nxgfv" Jul 13 22:01:09.531129 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:09.531103 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f4ff6c51-71c0-4f77-a4ef-ef5b602360fc-secret-grpc-tls\") pod \"thanos-querier-b49b6899d-nxgfv\" (UID: \"f4ff6c51-71c0-4f77-a4ef-ef5b602360fc\") " pod="openshift-monitoring/thanos-querier-b49b6899d-nxgfv" Jul 13 22:01:09.536322 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:09.536303 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftvs5\" (UniqueName: \"kubernetes.io/projected/f4ff6c51-71c0-4f77-a4ef-ef5b602360fc-kube-api-access-ftvs5\") pod \"thanos-querier-b49b6899d-nxgfv\" (UID: \"f4ff6c51-71c0-4f77-a4ef-ef5b602360fc\") " pod="openshift-monitoring/thanos-querier-b49b6899d-nxgfv" Jul 13 22:01:09.619516 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:09.619425 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-b49b6899d-nxgfv" Jul 13 22:01:09.755816 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:09.755700 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-b49b6899d-nxgfv"] Jul 13 22:01:09.758280 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:01:09.758255 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4ff6c51_71c0_4f77_a4ef_ef5b602360fc.slice/crio-483f01bf725fc474878c1cc7abf0d433de41dc59a9d1fdac4717298d2bba12ca WatchSource:0}: Error finding container 483f01bf725fc474878c1cc7abf0d433de41dc59a9d1fdac4717298d2bba12ca: Status 404 returned error can't find the container with id 483f01bf725fc474878c1cc7abf0d433de41dc59a9d1fdac4717298d2bba12ca Jul 13 22:01:09.884490 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:09.884449 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b49b6899d-nxgfv" event={"ID":"f4ff6c51-71c0-4f77-a4ef-ef5b602360fc","Type":"ContainerStarted","Data":"483f01bf725fc474878c1cc7abf0d433de41dc59a9d1fdac4717298d2bba12ca"} Jul 13 22:01:09.886499 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:09.886476 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5klkq" event={"ID":"3982d16e-b33e-4fc4-b834-bd8f9b6a35ce","Type":"ContainerStarted","Data":"9031ca5a0b5b01c79d50271222b5b43f65162254af6b24a4264ee46c5c00e505"} Jul 13 22:01:09.886617 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:09.886524 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5klkq" event={"ID":"3982d16e-b33e-4fc4-b834-bd8f9b6a35ce","Type":"ContainerStarted","Data":"0266d13aaeed7ae99e97baa59a27fc1c0c742eb24fa4c30f9cd45d5197842db2"} Jul 13 22:01:09.905571 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:09.905527 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-5klkq" podStartSLOduration=2.873192786 podStartE2EDuration="3.905511409s" totalImagesPullingTime="962.849033ms" totalInitContainerRuntime="69.46959ms" isStatefulPod=true podCreationTimestamp="2026-07-13 22:01:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:01:09.903853839 +0000 UTC m=+75.897608152" watchObservedRunningTime="2026-07-13 22:01:09.905511409 +0000 UTC m=+75.899265701" Jul 13 22:01:10.460831 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:10.460801 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6ddff77d76-7764x"] Jul 13 22:01:10.464433 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:10.464409 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6ddff77d76-7764x" Jul 13 22:01:10.469218 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:10.469175 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-ckkoh5a0jnfe7\"" Jul 13 22:01:10.469327 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:10.469217 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 13 22:01:10.469396 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:10.469362 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-fht7c\"" Jul 13 22:01:10.469476 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:10.469459 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 13 22:01:10.469788 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:10.469579 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 13 22:01:10.469788 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:10.469646 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 13 22:01:10.477892 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:10.477836 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6ddff77d76-7764x"] Jul 13 22:01:10.536118 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:10.536081 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmbxq\" (UniqueName: \"kubernetes.io/projected/b9b7c5ca-fc37-4ffc-978d-a555355b02be-kube-api-access-lmbxq\") pod \"metrics-server-6ddff77d76-7764x\" (UID: \"b9b7c5ca-fc37-4ffc-978d-a555355b02be\") " pod="openshift-monitoring/metrics-server-6ddff77d76-7764x" Jul 13 22:01:10.536297 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:10.536134 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b9b7c5ca-fc37-4ffc-978d-a555355b02be-audit-log\") pod \"metrics-server-6ddff77d76-7764x\" (UID: \"b9b7c5ca-fc37-4ffc-978d-a555355b02be\") " pod="openshift-monitoring/metrics-server-6ddff77d76-7764x" Jul 13 22:01:10.536297 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:10.536252 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b9b7c5ca-fc37-4ffc-978d-a555355b02be-secret-metrics-server-client-certs\") pod \"metrics-server-6ddff77d76-7764x\" (UID: \"b9b7c5ca-fc37-4ffc-978d-a555355b02be\") " pod="openshift-monitoring/metrics-server-6ddff77d76-7764x" Jul 13 22:01:10.536413 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:10.536299 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9b7c5ca-fc37-4ffc-978d-a555355b02be-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6ddff77d76-7764x\" (UID: \"b9b7c5ca-fc37-4ffc-978d-a555355b02be\") " pod="openshift-monitoring/metrics-server-6ddff77d76-7764x" Jul 13 22:01:10.536413 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:10.536337 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b9b7c5ca-fc37-4ffc-978d-a555355b02be-secret-metrics-server-tls\") pod \"metrics-server-6ddff77d76-7764x\" (UID: \"b9b7c5ca-fc37-4ffc-978d-a555355b02be\") " pod="openshift-monitoring/metrics-server-6ddff77d76-7764x" Jul 13 22:01:10.536413 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:10.536360 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b9b7c5ca-fc37-4ffc-978d-a555355b02be-client-ca-bundle\") pod \"metrics-server-6ddff77d76-7764x\" (UID: \"b9b7c5ca-fc37-4ffc-978d-a555355b02be\") " pod="openshift-monitoring/metrics-server-6ddff77d76-7764x" Jul 13 22:01:10.536413 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:10.536385 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b9b7c5ca-fc37-4ffc-978d-a555355b02be-metrics-server-audit-profiles\") pod \"metrics-server-6ddff77d76-7764x\" (UID: \"b9b7c5ca-fc37-4ffc-978d-a555355b02be\") " pod="openshift-monitoring/metrics-server-6ddff77d76-7764x" Jul 13 22:01:10.637572 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:10.637534 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmbxq\" (UniqueName: \"kubernetes.io/projected/b9b7c5ca-fc37-4ffc-978d-a555355b02be-kube-api-access-lmbxq\") pod \"metrics-server-6ddff77d76-7764x\" (UID: \"b9b7c5ca-fc37-4ffc-978d-a555355b02be\") " pod="openshift-monitoring/metrics-server-6ddff77d76-7764x" Jul 13 22:01:10.637748 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:10.637592 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b9b7c5ca-fc37-4ffc-978d-a555355b02be-audit-log\") pod \"metrics-server-6ddff77d76-7764x\" (UID: \"b9b7c5ca-fc37-4ffc-978d-a555355b02be\") " pod="openshift-monitoring/metrics-server-6ddff77d76-7764x" Jul 13 22:01:10.637748 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:10.637634 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b9b7c5ca-fc37-4ffc-978d-a555355b02be-secret-metrics-server-client-certs\") pod \"metrics-server-6ddff77d76-7764x\" (UID: \"b9b7c5ca-fc37-4ffc-978d-a555355b02be\") " pod="openshift-monitoring/metrics-server-6ddff77d76-7764x" Jul 13 22:01:10.637748 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:10.637656 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9b7c5ca-fc37-4ffc-978d-a555355b02be-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6ddff77d76-7764x\" (UID: \"b9b7c5ca-fc37-4ffc-978d-a555355b02be\") " pod="openshift-monitoring/metrics-server-6ddff77d76-7764x" Jul 13 22:01:10.637748 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:10.637687 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b9b7c5ca-fc37-4ffc-978d-a555355b02be-secret-metrics-server-tls\") pod \"metrics-server-6ddff77d76-7764x\" (UID: \"b9b7c5ca-fc37-4ffc-978d-a555355b02be\") " pod="openshift-monitoring/metrics-server-6ddff77d76-7764x" Jul 13 22:01:10.637748 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:10.637716 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b9b7c5ca-fc37-4ffc-978d-a555355b02be-client-ca-bundle\") pod \"metrics-server-6ddff77d76-7764x\" (UID: \"b9b7c5ca-fc37-4ffc-978d-a555355b02be\") " pod="openshift-monitoring/metrics-server-6ddff77d76-7764x" Jul 13 22:01:10.637748 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:10.637741 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b9b7c5ca-fc37-4ffc-978d-a555355b02be-metrics-server-audit-profiles\") pod \"metrics-server-6ddff77d76-7764x\" (UID: \"b9b7c5ca-fc37-4ffc-978d-a555355b02be\") " pod="openshift-monitoring/metrics-server-6ddff77d76-7764x" Jul 13 22:01:10.638477 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:10.638448 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b9b7c5ca-fc37-4ffc-978d-a555355b02be-audit-log\") pod \"metrics-server-6ddff77d76-7764x\" (UID: \"b9b7c5ca-fc37-4ffc-978d-a555355b02be\") " pod="openshift-monitoring/metrics-server-6ddff77d76-7764x" Jul 13 22:01:10.638966 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:10.638931 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9b7c5ca-fc37-4ffc-978d-a555355b02be-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6ddff77d76-7764x\" (UID: \"b9b7c5ca-fc37-4ffc-978d-a555355b02be\") " pod="openshift-monitoring/metrics-server-6ddff77d76-7764x" Jul 13 22:01:10.639121 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:10.639094 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b9b7c5ca-fc37-4ffc-978d-a555355b02be-metrics-server-audit-profiles\") pod \"metrics-server-6ddff77d76-7764x\" (UID: \"b9b7c5ca-fc37-4ffc-978d-a555355b02be\") " pod="openshift-monitoring/metrics-server-6ddff77d76-7764x" Jul 13 22:01:10.640625 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:10.640601 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b9b7c5ca-fc37-4ffc-978d-a555355b02be-secret-metrics-server-client-certs\") pod \"metrics-server-6ddff77d76-7764x\" (UID: \"b9b7c5ca-fc37-4ffc-978d-a555355b02be\") " pod="openshift-monitoring/metrics-server-6ddff77d76-7764x" Jul 13 22:01:10.640779 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:10.640760 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b9b7c5ca-fc37-4ffc-978d-a555355b02be-secret-metrics-server-tls\") pod \"metrics-server-6ddff77d76-7764x\" (UID: \"b9b7c5ca-fc37-4ffc-978d-a555355b02be\") " pod="openshift-monitoring/metrics-server-6ddff77d76-7764x" Jul 13 22:01:10.640838 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:10.640766 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b9b7c5ca-fc37-4ffc-978d-a555355b02be-client-ca-bundle\") pod \"metrics-server-6ddff77d76-7764x\" (UID: \"b9b7c5ca-fc37-4ffc-978d-a555355b02be\") " pod="openshift-monitoring/metrics-server-6ddff77d76-7764x" Jul 13 22:01:10.646615 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:10.646586 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmbxq\" (UniqueName: \"kubernetes.io/projected/b9b7c5ca-fc37-4ffc-978d-a555355b02be-kube-api-access-lmbxq\") pod \"metrics-server-6ddff77d76-7764x\" (UID: \"b9b7c5ca-fc37-4ffc-978d-a555355b02be\") " pod="openshift-monitoring/metrics-server-6ddff77d76-7764x" Jul 13 22:01:10.775831 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:10.775743 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6ddff77d76-7764x" Jul 13 22:01:10.949229 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:10.948863 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6ddff77d76-7764x"] Jul 13 22:01:10.952895 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:01:10.952864 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9b7c5ca_fc37_4ffc_978d_a555355b02be.slice/crio-348898023a5a76a17494c65022670507694063abb93e74832dd74d3aa6e88253 WatchSource:0}: Error finding container 348898023a5a76a17494c65022670507694063abb93e74832dd74d3aa6e88253: Status 404 returned error can't find the container with id 348898023a5a76a17494c65022670507694063abb93e74832dd74d3aa6e88253 Jul 13 22:01:11.898558 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:11.898524 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6ddff77d76-7764x" event={"ID":"b9b7c5ca-fc37-4ffc-978d-a555355b02be","Type":"ContainerStarted","Data":"348898023a5a76a17494c65022670507694063abb93e74832dd74d3aa6e88253"} Jul 13 22:01:11.901312 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:11.901286 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b49b6899d-nxgfv" event={"ID":"f4ff6c51-71c0-4f77-a4ef-ef5b602360fc","Type":"ContainerStarted","Data":"115ac79e271bc57f46a2de206fad099002d0eed7b68cfbb41719d8d61233062d"} Jul 13 22:01:11.901406 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:11.901359 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b49b6899d-nxgfv" event={"ID":"f4ff6c51-71c0-4f77-a4ef-ef5b602360fc","Type":"ContainerStarted","Data":"64099b0369808b618f05e38cf87df545843fb905379fdbd31adf7f7b35f0098d"} Jul 13 22:01:11.901406 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:11.901401 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b49b6899d-nxgfv" event={"ID":"f4ff6c51-71c0-4f77-a4ef-ef5b602360fc","Type":"ContainerStarted","Data":"4ab6f5f7dadb652786a17fc36b6272978acfa4ba4b46281596a8433467469773"} Jul 13 22:01:12.908022 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:12.907980 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6ddff77d76-7764x" event={"ID":"b9b7c5ca-fc37-4ffc-978d-a555355b02be","Type":"ContainerStarted","Data":"4e6abe16e3d2df024ef4bd536558951e08367232b9e03d24ab2db0935d102630"} Jul 13 22:01:12.933468 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:12.933413 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6ddff77d76-7764x" podStartSLOduration=1.1880129560000001 podStartE2EDuration="2.933396376s" totalImagesPullingTime="1.74538342s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:01:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:01:12.931886542 +0000 UTC m=+78.925640835" watchObservedRunningTime="2026-07-13 22:01:12.933396376 +0000 UTC m=+78.927150668" Jul 13 22:01:13.918499 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:13.918460 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b49b6899d-nxgfv" event={"ID":"f4ff6c51-71c0-4f77-a4ef-ef5b602360fc","Type":"ContainerStarted","Data":"c2c913fd055a01171a9f10e1dab784f14ee594feb7b7e838f62acc62f000ca7d"} Jul 13 22:01:13.918879 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:13.918554 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b49b6899d-nxgfv" event={"ID":"f4ff6c51-71c0-4f77-a4ef-ef5b602360fc","Type":"ContainerStarted","Data":"24b459b18bc5e50338004efa0a1d241826cbdc71bc2336e54089872d64317693"} Jul 13 22:01:13.918879 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:13.918583 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b49b6899d-nxgfv" event={"ID":"f4ff6c51-71c0-4f77-a4ef-ef5b602360fc","Type":"ContainerStarted","Data":"086a0f141a9ac1a55c94b56dce9105781fa066380776d999f94e41a18f6a72f8"} Jul 13 22:01:13.946957 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:13.946908 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-b49b6899d-nxgfv" podStartSLOduration=1.807216286 podStartE2EDuration="4.946892727s" totalImagesPullingTime="3.139676441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:01:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:01:13.946537645 +0000 UTC m=+79.940291975" watchObservedRunningTime="2026-07-13 22:01:13.946892727 +0000 UTC m=+79.940647018" Jul 13 22:01:14.922020 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:14.921914 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-b49b6899d-nxgfv" Jul 13 22:01:20.931153 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:20.931124 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-b49b6899d-nxgfv" Jul 13 22:01:30.776850 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:30.776807 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6ddff77d76-7764x" Jul 13 22:01:30.777332 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:30.776887 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6ddff77d76-7764x" Jul 13 22:01:36.870027 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:36.870000 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-rg8tn" Jul 13 22:01:50.781908 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:50.781872 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6ddff77d76-7764x" Jul 13 22:01:50.785649 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:01:50.785628 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6ddff77d76-7764x" Jul 13 22:03:27.423769 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:03:27.423728 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vgbhv"] Jul 13 22:03:27.425951 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:03:27.425930 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vgbhv" Jul 13 22:03:27.428490 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:03:27.428461 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 13 22:03:27.428589 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:03:27.428464 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-t6xk6\"" Jul 13 22:03:27.429506 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:03:27.429485 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 13 22:03:27.440313 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:03:27.440291 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vgbhv"] Jul 13 22:03:27.527616 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:03:27.527579 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6a7f3103-4261-48e6-b156-77408231110c-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vgbhv\" (UID: \"6a7f3103-4261-48e6-b156-77408231110c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vgbhv" Jul 13 22:03:27.527798 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:03:27.527637 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6a7f3103-4261-48e6-b156-77408231110c-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vgbhv\" (UID: \"6a7f3103-4261-48e6-b156-77408231110c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vgbhv" Jul 13 22:03:27.527798 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:03:27.527686 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxfwj\" (UniqueName: \"kubernetes.io/projected/6a7f3103-4261-48e6-b156-77408231110c-kube-api-access-pxfwj\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vgbhv\" (UID: \"6a7f3103-4261-48e6-b156-77408231110c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vgbhv" Jul 13 22:03:27.628026 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:03:27.627981 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6a7f3103-4261-48e6-b156-77408231110c-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vgbhv\" (UID: \"6a7f3103-4261-48e6-b156-77408231110c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vgbhv" Jul 13 22:03:27.628169 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:03:27.628039 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxfwj\" (UniqueName: \"kubernetes.io/projected/6a7f3103-4261-48e6-b156-77408231110c-kube-api-access-pxfwj\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vgbhv\" (UID: \"6a7f3103-4261-48e6-b156-77408231110c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vgbhv" Jul 13 22:03:27.628169 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:03:27.628092 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6a7f3103-4261-48e6-b156-77408231110c-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vgbhv\" (UID: \"6a7f3103-4261-48e6-b156-77408231110c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vgbhv" Jul 13 22:03:27.628389 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:03:27.628370 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6a7f3103-4261-48e6-b156-77408231110c-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vgbhv\" (UID: \"6a7f3103-4261-48e6-b156-77408231110c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vgbhv" Jul 13 22:03:27.628425 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:03:27.628405 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6a7f3103-4261-48e6-b156-77408231110c-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vgbhv\" (UID: \"6a7f3103-4261-48e6-b156-77408231110c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vgbhv" Jul 13 22:03:27.645481 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:03:27.645443 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxfwj\" (UniqueName: \"kubernetes.io/projected/6a7f3103-4261-48e6-b156-77408231110c-kube-api-access-pxfwj\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vgbhv\" (UID: \"6a7f3103-4261-48e6-b156-77408231110c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vgbhv" Jul 13 22:03:27.734628 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:03:27.734529 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vgbhv" Jul 13 22:03:27.866653 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:03:27.866632 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vgbhv"] Jul 13 22:03:27.869466 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:03:27.869441 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a7f3103_4261_48e6_b156_77408231110c.slice/crio-a9abd2d329c763b2ce6952cb75a07096582e9bab2e341fa7ba7750de558dbc9b WatchSource:0}: Error finding container a9abd2d329c763b2ce6952cb75a07096582e9bab2e341fa7ba7750de558dbc9b: Status 404 returned error can't find the container with id a9abd2d329c763b2ce6952cb75a07096582e9bab2e341fa7ba7750de558dbc9b Jul 13 22:03:28.370028 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:03:28.369988 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vgbhv" event={"ID":"6a7f3103-4261-48e6-b156-77408231110c","Type":"ContainerStarted","Data":"a9abd2d329c763b2ce6952cb75a07096582e9bab2e341fa7ba7750de558dbc9b"} Jul 13 22:03:37.406636 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:03:37.406598 2555 generic.go:356] "Generic (PLEG): container finished" podID="6a7f3103-4261-48e6-b156-77408231110c" containerID="65c7bed00bd5d59f16f0e3cae8a8ae83203ee5db14d40e101ee4db274c47b5c5" exitCode=0 Jul 13 22:03:37.407058 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:03:37.406686 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vgbhv" event={"ID":"6a7f3103-4261-48e6-b156-77408231110c","Type":"ContainerDied","Data":"65c7bed00bd5d59f16f0e3cae8a8ae83203ee5db14d40e101ee4db274c47b5c5"} Jul 13 22:03:39.415836 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:03:39.415805 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vgbhv" event={"ID":"6a7f3103-4261-48e6-b156-77408231110c","Type":"ContainerStarted","Data":"2de30539bd2826d6eeab770b49d85e579c25e096e98c075d8c1961a5697600d8"} Jul 13 22:03:40.421257 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:03:40.421185 2555 generic.go:356] "Generic (PLEG): container finished" podID="6a7f3103-4261-48e6-b156-77408231110c" containerID="2de30539bd2826d6eeab770b49d85e579c25e096e98c075d8c1961a5697600d8" exitCode=0 Jul 13 22:03:40.421633 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:03:40.421271 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vgbhv" event={"ID":"6a7f3103-4261-48e6-b156-77408231110c","Type":"ContainerDied","Data":"2de30539bd2826d6eeab770b49d85e579c25e096e98c075d8c1961a5697600d8"} Jul 13 22:03:49.456057 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:03:49.456022 2555 generic.go:356] "Generic (PLEG): container finished" podID="6a7f3103-4261-48e6-b156-77408231110c" containerID="d82f606de6e47e6e720c4b35a4e9a48f175cd6cc32c5fda07d9d6e4892f6837f" exitCode=0 Jul 13 22:03:49.456472 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:03:49.456096 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vgbhv" event={"ID":"6a7f3103-4261-48e6-b156-77408231110c","Type":"ContainerDied","Data":"d82f606de6e47e6e720c4b35a4e9a48f175cd6cc32c5fda07d9d6e4892f6837f"} Jul 13 22:03:50.618704 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:03:50.618667 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vgbhv" Jul 13 22:03:50.710301 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:03:50.710265 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6a7f3103-4261-48e6-b156-77408231110c-bundle\") pod \"6a7f3103-4261-48e6-b156-77408231110c\" (UID: \"6a7f3103-4261-48e6-b156-77408231110c\") " Jul 13 22:03:50.710519 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:03:50.710336 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6a7f3103-4261-48e6-b156-77408231110c-util\") pod \"6a7f3103-4261-48e6-b156-77408231110c\" (UID: \"6a7f3103-4261-48e6-b156-77408231110c\") " Jul 13 22:03:50.710519 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:03:50.710389 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pxfwj\" (UniqueName: \"kubernetes.io/projected/6a7f3103-4261-48e6-b156-77408231110c-kube-api-access-pxfwj\") pod \"6a7f3103-4261-48e6-b156-77408231110c\" (UID: \"6a7f3103-4261-48e6-b156-77408231110c\") " Jul 13 22:03:50.710963 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:03:50.710940 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a7f3103-4261-48e6-b156-77408231110c-bundle" (OuterVolumeSpecName: "bundle") pod "6a7f3103-4261-48e6-b156-77408231110c" (UID: "6a7f3103-4261-48e6-b156-77408231110c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:03:50.712705 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:03:50.712680 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a7f3103-4261-48e6-b156-77408231110c-kube-api-access-pxfwj" (OuterVolumeSpecName: "kube-api-access-pxfwj") pod "6a7f3103-4261-48e6-b156-77408231110c" (UID: "6a7f3103-4261-48e6-b156-77408231110c"). InnerVolumeSpecName "kube-api-access-pxfwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:03:50.715690 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:03:50.715666 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a7f3103-4261-48e6-b156-77408231110c-util" (OuterVolumeSpecName: "util") pod "6a7f3103-4261-48e6-b156-77408231110c" (UID: "6a7f3103-4261-48e6-b156-77408231110c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:03:50.811579 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:03:50.811546 2555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6a7f3103-4261-48e6-b156-77408231110c-bundle\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:03:50.811579 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:03:50.811573 2555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6a7f3103-4261-48e6-b156-77408231110c-util\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:03:50.811579 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:03:50.811583 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxfwj\" (UniqueName: \"kubernetes.io/projected/6a7f3103-4261-48e6-b156-77408231110c-kube-api-access-pxfwj\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:03:51.465308 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:03:51.465277 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vgbhv" Jul 13 22:03:51.465308 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:03:51.465299 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vgbhv" event={"ID":"6a7f3103-4261-48e6-b156-77408231110c","Type":"ContainerDied","Data":"a9abd2d329c763b2ce6952cb75a07096582e9bab2e341fa7ba7750de558dbc9b"} Jul 13 22:03:51.465507 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:03:51.465342 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9abd2d329c763b2ce6952cb75a07096582e9bab2e341fa7ba7750de558dbc9b" Jul 13 22:04:00.586477 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:00.586442 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvrwvn"] Jul 13 22:04:00.586873 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:00.586679 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a7f3103-4261-48e6-b156-77408231110c" containerName="pull" Jul 13 22:04:00.586873 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:00.586690 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a7f3103-4261-48e6-b156-77408231110c" containerName="pull" Jul 13 22:04:00.586873 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:00.586698 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a7f3103-4261-48e6-b156-77408231110c" containerName="util" Jul 13 22:04:00.586873 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:00.586704 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a7f3103-4261-48e6-b156-77408231110c" containerName="util" Jul 13 22:04:00.586873 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:00.586715 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a7f3103-4261-48e6-b156-77408231110c" containerName="extract" Jul 13 22:04:00.586873 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:00.586720 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a7f3103-4261-48e6-b156-77408231110c" containerName="extract" Jul 13 22:04:00.586873 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:00.586764 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a7f3103-4261-48e6-b156-77408231110c" containerName="extract" Jul 13 22:04:00.589848 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:00.589830 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvrwvn" Jul 13 22:04:00.592341 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:00.592317 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 13 22:04:00.592441 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:00.592364 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-t6xk6\"" Jul 13 22:04:00.592441 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:00.592392 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 13 22:04:00.598650 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:00.598630 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvrwvn"] Jul 13 22:04:00.688956 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:00.688922 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fb6b74f3-9256-48a9-af8e-57f5d7e55521-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvrwvn\" (UID: \"fb6b74f3-9256-48a9-af8e-57f5d7e55521\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvrwvn" Jul 13 22:04:00.689142 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:00.688976 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rk68\" (UniqueName: \"kubernetes.io/projected/fb6b74f3-9256-48a9-af8e-57f5d7e55521-kube-api-access-6rk68\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvrwvn\" (UID: \"fb6b74f3-9256-48a9-af8e-57f5d7e55521\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvrwvn" Jul 13 22:04:00.689142 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:00.689005 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fb6b74f3-9256-48a9-af8e-57f5d7e55521-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvrwvn\" (UID: \"fb6b74f3-9256-48a9-af8e-57f5d7e55521\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvrwvn" Jul 13 22:04:00.789784 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:00.789750 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fb6b74f3-9256-48a9-af8e-57f5d7e55521-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvrwvn\" (UID: \"fb6b74f3-9256-48a9-af8e-57f5d7e55521\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvrwvn" Jul 13 22:04:00.789956 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:00.789800 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rk68\" (UniqueName: \"kubernetes.io/projected/fb6b74f3-9256-48a9-af8e-57f5d7e55521-kube-api-access-6rk68\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvrwvn\" (UID: \"fb6b74f3-9256-48a9-af8e-57f5d7e55521\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvrwvn" Jul 13 22:04:00.789956 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:00.789828 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fb6b74f3-9256-48a9-af8e-57f5d7e55521-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvrwvn\" (UID: \"fb6b74f3-9256-48a9-af8e-57f5d7e55521\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvrwvn" Jul 13 22:04:00.790247 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:00.790228 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fb6b74f3-9256-48a9-af8e-57f5d7e55521-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvrwvn\" (UID: \"fb6b74f3-9256-48a9-af8e-57f5d7e55521\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvrwvn" Jul 13 22:04:00.790247 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:00.790239 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fb6b74f3-9256-48a9-af8e-57f5d7e55521-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvrwvn\" (UID: \"fb6b74f3-9256-48a9-af8e-57f5d7e55521\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvrwvn" Jul 13 22:04:00.799159 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:00.799127 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rk68\" (UniqueName: \"kubernetes.io/projected/fb6b74f3-9256-48a9-af8e-57f5d7e55521-kube-api-access-6rk68\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvrwvn\" (UID: \"fb6b74f3-9256-48a9-af8e-57f5d7e55521\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvrwvn" Jul 13 22:04:00.898316 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:00.898287 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvrwvn" Jul 13 22:04:01.025755 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:01.025724 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvrwvn"] Jul 13 22:04:01.029016 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:04:01.028988 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb6b74f3_9256_48a9_af8e_57f5d7e55521.slice/crio-971cab2742c5f1959559e5e3a0074f85f785ebbab8605db492448800762e9759 WatchSource:0}: Error finding container 971cab2742c5f1959559e5e3a0074f85f785ebbab8605db492448800762e9759: Status 404 returned error can't find the container with id 971cab2742c5f1959559e5e3a0074f85f785ebbab8605db492448800762e9759 Jul 13 22:04:01.502658 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:01.502618 2555 generic.go:356] "Generic (PLEG): container finished" podID="fb6b74f3-9256-48a9-af8e-57f5d7e55521" containerID="2b77714cac65a59a219f7f61cde9a23681b0bbbad0e4ea1011820f86c898c4a5" exitCode=0 Jul 13 22:04:01.502818 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:01.502684 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvrwvn" event={"ID":"fb6b74f3-9256-48a9-af8e-57f5d7e55521","Type":"ContainerDied","Data":"2b77714cac65a59a219f7f61cde9a23681b0bbbad0e4ea1011820f86c898c4a5"} Jul 13 22:04:01.502818 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:01.502723 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvrwvn" event={"ID":"fb6b74f3-9256-48a9-af8e-57f5d7e55521","Type":"ContainerStarted","Data":"971cab2742c5f1959559e5e3a0074f85f785ebbab8605db492448800762e9759"} Jul 13 22:04:06.736689 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:06.736653 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-j2mwk"] Jul 13 22:04:06.738702 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:06.738685 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-j2mwk" Jul 13 22:04:06.741107 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:06.741087 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 13 22:04:06.741215 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:06.741109 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 13 22:04:06.742008 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:06.741989 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-6jwzk\"" Jul 13 22:04:06.750829 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:06.750808 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-j2mwk"] Jul 13 22:04:06.835230 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:06.835155 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e7ff1495-0eb8-4d56-83c4-98da00bf0b75-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-j2mwk\" (UID: \"e7ff1495-0eb8-4d56-83c4-98da00bf0b75\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-j2mwk" Jul 13 22:04:06.835230 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:06.835237 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvckw\" (UniqueName: \"kubernetes.io/projected/e7ff1495-0eb8-4d56-83c4-98da00bf0b75-kube-api-access-zvckw\") pod \"cert-manager-cainjector-559cdd8c87-j2mwk\" (UID: \"e7ff1495-0eb8-4d56-83c4-98da00bf0b75\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-j2mwk" Jul 13 22:04:06.936010 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:06.935969 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e7ff1495-0eb8-4d56-83c4-98da00bf0b75-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-j2mwk\" (UID: \"e7ff1495-0eb8-4d56-83c4-98da00bf0b75\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-j2mwk" Jul 13 22:04:06.936010 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:06.936015 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvckw\" (UniqueName: \"kubernetes.io/projected/e7ff1495-0eb8-4d56-83c4-98da00bf0b75-kube-api-access-zvckw\") pod \"cert-manager-cainjector-559cdd8c87-j2mwk\" (UID: \"e7ff1495-0eb8-4d56-83c4-98da00bf0b75\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-j2mwk" Jul 13 22:04:06.945575 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:06.945551 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e7ff1495-0eb8-4d56-83c4-98da00bf0b75-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-j2mwk\" (UID: \"e7ff1495-0eb8-4d56-83c4-98da00bf0b75\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-j2mwk" Jul 13 22:04:06.945695 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:06.945624 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvckw\" (UniqueName: \"kubernetes.io/projected/e7ff1495-0eb8-4d56-83c4-98da00bf0b75-kube-api-access-zvckw\") pod \"cert-manager-cainjector-559cdd8c87-j2mwk\" (UID: \"e7ff1495-0eb8-4d56-83c4-98da00bf0b75\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-j2mwk" Jul 13 22:04:07.048009 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:07.047914 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-j2mwk" Jul 13 22:04:07.176420 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:07.176361 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-j2mwk"] Jul 13 22:04:07.179137 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:04:07.179112 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode7ff1495_0eb8_4d56_83c4_98da00bf0b75.slice/crio-b82aa6e39c3e037c7bb280e1b17def7888339eb1ba06133d3808c737469a36f7 WatchSource:0}: Error finding container b82aa6e39c3e037c7bb280e1b17def7888339eb1ba06133d3808c737469a36f7: Status 404 returned error can't find the container with id b82aa6e39c3e037c7bb280e1b17def7888339eb1ba06133d3808c737469a36f7 Jul 13 22:04:07.525247 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:07.525191 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-j2mwk" event={"ID":"e7ff1495-0eb8-4d56-83c4-98da00bf0b75","Type":"ContainerStarted","Data":"b82aa6e39c3e037c7bb280e1b17def7888339eb1ba06133d3808c737469a36f7"} Jul 13 22:04:10.539875 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:10.539834 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-j2mwk" event={"ID":"e7ff1495-0eb8-4d56-83c4-98da00bf0b75","Type":"ContainerStarted","Data":"ccdad49806e368530ee5fe02cd2ee658195d24d4200ce38ca80c905b3be26b06"} Jul 13 22:04:10.556816 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:10.556768 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-j2mwk" podStartSLOduration=1.630596276 podStartE2EDuration="4.55675054s" totalImagesPullingTime="2.926154264s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:04:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:04:10.555734268 +0000 UTC m=+256.549488563" watchObservedRunningTime="2026-07-13 22:04:10.55675054 +0000 UTC m=+256.550504832" Jul 13 22:04:17.565258 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:17.565227 2555 generic.go:356] "Generic (PLEG): container finished" podID="fb6b74f3-9256-48a9-af8e-57f5d7e55521" containerID="81ae285584ee8e80e8d60cd5be4c97ee3ffd45b2ff2dc794f1141a1e334d880b" exitCode=0 Jul 13 22:04:17.565689 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:17.565317 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvrwvn" event={"ID":"fb6b74f3-9256-48a9-af8e-57f5d7e55521","Type":"ContainerDied","Data":"81ae285584ee8e80e8d60cd5be4c97ee3ffd45b2ff2dc794f1141a1e334d880b"} Jul 13 22:04:18.570093 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:18.570057 2555 generic.go:356] "Generic (PLEG): container finished" podID="fb6b74f3-9256-48a9-af8e-57f5d7e55521" containerID="6e9e3cd3040d81803849f3e51eec1fafb385e249dc7872db5dc8b774aff08e4a" exitCode=0 Jul 13 22:04:18.570499 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:18.570105 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvrwvn" event={"ID":"fb6b74f3-9256-48a9-af8e-57f5d7e55521","Type":"ContainerDied","Data":"6e9e3cd3040d81803849f3e51eec1fafb385e249dc7872db5dc8b774aff08e4a"} Jul 13 22:04:19.714390 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:19.714357 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvrwvn" Jul 13 22:04:19.837788 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:19.837691 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6rk68\" (UniqueName: \"kubernetes.io/projected/fb6b74f3-9256-48a9-af8e-57f5d7e55521-kube-api-access-6rk68\") pod \"fb6b74f3-9256-48a9-af8e-57f5d7e55521\" (UID: \"fb6b74f3-9256-48a9-af8e-57f5d7e55521\") " Jul 13 22:04:19.837788 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:19.837746 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fb6b74f3-9256-48a9-af8e-57f5d7e55521-util\") pod \"fb6b74f3-9256-48a9-af8e-57f5d7e55521\" (UID: \"fb6b74f3-9256-48a9-af8e-57f5d7e55521\") " Jul 13 22:04:19.837788 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:19.837772 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fb6b74f3-9256-48a9-af8e-57f5d7e55521-bundle\") pod \"fb6b74f3-9256-48a9-af8e-57f5d7e55521\" (UID: \"fb6b74f3-9256-48a9-af8e-57f5d7e55521\") " Jul 13 22:04:19.838240 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:19.838183 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb6b74f3-9256-48a9-af8e-57f5d7e55521-bundle" (OuterVolumeSpecName: "bundle") pod "fb6b74f3-9256-48a9-af8e-57f5d7e55521" (UID: "fb6b74f3-9256-48a9-af8e-57f5d7e55521"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:04:19.839985 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:19.839964 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb6b74f3-9256-48a9-af8e-57f5d7e55521-kube-api-access-6rk68" (OuterVolumeSpecName: "kube-api-access-6rk68") pod "fb6b74f3-9256-48a9-af8e-57f5d7e55521" (UID: "fb6b74f3-9256-48a9-af8e-57f5d7e55521"). InnerVolumeSpecName "kube-api-access-6rk68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:04:19.842964 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:19.842941 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb6b74f3-9256-48a9-af8e-57f5d7e55521-util" (OuterVolumeSpecName: "util") pod "fb6b74f3-9256-48a9-af8e-57f5d7e55521" (UID: "fb6b74f3-9256-48a9-af8e-57f5d7e55521"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:04:19.939353 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:19.939315 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rk68\" (UniqueName: \"kubernetes.io/projected/fb6b74f3-9256-48a9-af8e-57f5d7e55521-kube-api-access-6rk68\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:04:19.939353 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:19.939346 2555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fb6b74f3-9256-48a9-af8e-57f5d7e55521-util\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:04:19.939353 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:19.939356 2555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fb6b74f3-9256-48a9-af8e-57f5d7e55521-bundle\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:04:20.578641 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:20.578605 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvrwvn" event={"ID":"fb6b74f3-9256-48a9-af8e-57f5d7e55521","Type":"ContainerDied","Data":"971cab2742c5f1959559e5e3a0074f85f785ebbab8605db492448800762e9759"} Jul 13 22:04:20.578641 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:20.578641 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvrwvn" Jul 13 22:04:20.578848 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:20.578660 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="971cab2742c5f1959559e5e3a0074f85f785ebbab8605db492448800762e9759" Jul 13 22:04:25.538785 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:25.538751 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-blmfj"] Jul 13 22:04:25.539177 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:25.539013 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb6b74f3-9256-48a9-af8e-57f5d7e55521" containerName="extract" Jul 13 22:04:25.539177 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:25.539023 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb6b74f3-9256-48a9-af8e-57f5d7e55521" containerName="extract" Jul 13 22:04:25.539177 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:25.539035 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb6b74f3-9256-48a9-af8e-57f5d7e55521" containerName="util" Jul 13 22:04:25.539177 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:25.539040 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb6b74f3-9256-48a9-af8e-57f5d7e55521" containerName="util" Jul 13 22:04:25.539177 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:25.539048 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb6b74f3-9256-48a9-af8e-57f5d7e55521" containerName="pull" Jul 13 22:04:25.539177 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:25.539053 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb6b74f3-9256-48a9-af8e-57f5d7e55521" containerName="pull" Jul 13 22:04:25.539177 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:25.539093 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb6b74f3-9256-48a9-af8e-57f5d7e55521" containerName="extract" Jul 13 22:04:25.541067 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:25.541052 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-blmfj" Jul 13 22:04:25.543768 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:25.543746 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 13 22:04:25.543768 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:25.543750 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 13 22:04:25.544517 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:25.544499 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-tdxkj\"" Jul 13 22:04:25.553010 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:25.552988 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-blmfj"] Jul 13 22:04:25.681169 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:25.681135 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjnkv\" (UniqueName: \"kubernetes.io/projected/1b01374e-3cab-4b97-9537-73535f0fd133-kube-api-access-xjnkv\") pod \"openshift-lws-operator-fd8ccff4c-blmfj\" (UID: \"1b01374e-3cab-4b97-9537-73535f0fd133\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-blmfj" Jul 13 22:04:25.681376 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:25.681196 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b01374e-3cab-4b97-9537-73535f0fd133-tmp\") pod \"openshift-lws-operator-fd8ccff4c-blmfj\" (UID: \"1b01374e-3cab-4b97-9537-73535f0fd133\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-blmfj" Jul 13 22:04:25.782015 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:25.781980 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b01374e-3cab-4b97-9537-73535f0fd133-tmp\") pod \"openshift-lws-operator-fd8ccff4c-blmfj\" (UID: \"1b01374e-3cab-4b97-9537-73535f0fd133\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-blmfj" Jul 13 22:04:25.782148 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:25.782036 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjnkv\" (UniqueName: \"kubernetes.io/projected/1b01374e-3cab-4b97-9537-73535f0fd133-kube-api-access-xjnkv\") pod \"openshift-lws-operator-fd8ccff4c-blmfj\" (UID: \"1b01374e-3cab-4b97-9537-73535f0fd133\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-blmfj" Jul 13 22:04:25.782438 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:25.782407 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b01374e-3cab-4b97-9537-73535f0fd133-tmp\") pod \"openshift-lws-operator-fd8ccff4c-blmfj\" (UID: \"1b01374e-3cab-4b97-9537-73535f0fd133\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-blmfj" Jul 13 22:04:25.790450 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:25.790400 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjnkv\" (UniqueName: \"kubernetes.io/projected/1b01374e-3cab-4b97-9537-73535f0fd133-kube-api-access-xjnkv\") pod \"openshift-lws-operator-fd8ccff4c-blmfj\" (UID: \"1b01374e-3cab-4b97-9537-73535f0fd133\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-blmfj" Jul 13 22:04:25.850534 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:25.850504 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-blmfj" Jul 13 22:04:25.978185 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:25.978162 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-blmfj"] Jul 13 22:04:25.980921 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:04:25.980893 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b01374e_3cab_4b97_9537_73535f0fd133.slice/crio-be1f890eb0ceb620e45da325e82282404736807105a5847436d06c485e6b50e5 WatchSource:0}: Error finding container be1f890eb0ceb620e45da325e82282404736807105a5847436d06c485e6b50e5: Status 404 returned error can't find the container with id be1f890eb0ceb620e45da325e82282404736807105a5847436d06c485e6b50e5 Jul 13 22:04:26.601792 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:26.601761 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-blmfj" event={"ID":"1b01374e-3cab-4b97-9537-73535f0fd133","Type":"ContainerStarted","Data":"be1f890eb0ceb620e45da325e82282404736807105a5847436d06c485e6b50e5"} Jul 13 22:04:28.611034 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:28.610997 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-blmfj" event={"ID":"1b01374e-3cab-4b97-9537-73535f0fd133","Type":"ContainerStarted","Data":"b5c344e95d3085b6c9fcf8e2e4b3b78658dcfdc51b78202fd07c01e194623b47"} Jul 13 22:04:28.628195 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:28.628135 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-blmfj" podStartSLOduration=1.7854449730000002 podStartE2EDuration="3.628114833s" totalImagesPullingTime="1.84266986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:04:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:04:28.62784826 +0000 UTC m=+274.621602553" watchObservedRunningTime="2026-07-13 22:04:28.628114833 +0000 UTC m=+274.621869127" Jul 13 22:04:30.723038 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:30.723004 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2v8g7"] Jul 13 22:04:30.725128 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:30.725112 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2v8g7" Jul 13 22:04:30.727393 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:30.727368 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 13 22:04:30.727682 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:30.727468 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 13 22:04:30.728406 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:30.728392 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-t6xk6\"" Jul 13 22:04:30.734289 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:30.734264 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2v8g7"] Jul 13 22:04:30.817762 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:30.817731 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgb98\" (UniqueName: \"kubernetes.io/projected/fd53a7ee-bdf6-4a6a-b57e-cd9c6b9c2822-kube-api-access-xgb98\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2v8g7\" (UID: \"fd53a7ee-bdf6-4a6a-b57e-cd9c6b9c2822\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2v8g7" Jul 13 22:04:30.817923 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:30.817774 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd53a7ee-bdf6-4a6a-b57e-cd9c6b9c2822-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2v8g7\" (UID: \"fd53a7ee-bdf6-4a6a-b57e-cd9c6b9c2822\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2v8g7" Jul 13 22:04:30.817923 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:30.817792 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd53a7ee-bdf6-4a6a-b57e-cd9c6b9c2822-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2v8g7\" (UID: \"fd53a7ee-bdf6-4a6a-b57e-cd9c6b9c2822\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2v8g7" Jul 13 22:04:30.921723 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:30.920593 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgb98\" (UniqueName: \"kubernetes.io/projected/fd53a7ee-bdf6-4a6a-b57e-cd9c6b9c2822-kube-api-access-xgb98\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2v8g7\" (UID: \"fd53a7ee-bdf6-4a6a-b57e-cd9c6b9c2822\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2v8g7" Jul 13 22:04:30.921723 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:30.920667 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd53a7ee-bdf6-4a6a-b57e-cd9c6b9c2822-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2v8g7\" (UID: \"fd53a7ee-bdf6-4a6a-b57e-cd9c6b9c2822\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2v8g7" Jul 13 22:04:30.921723 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:30.920703 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd53a7ee-bdf6-4a6a-b57e-cd9c6b9c2822-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2v8g7\" (UID: \"fd53a7ee-bdf6-4a6a-b57e-cd9c6b9c2822\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2v8g7" Jul 13 22:04:30.921723 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:30.921174 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd53a7ee-bdf6-4a6a-b57e-cd9c6b9c2822-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2v8g7\" (UID: \"fd53a7ee-bdf6-4a6a-b57e-cd9c6b9c2822\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2v8g7" Jul 13 22:04:30.922222 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:30.922179 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd53a7ee-bdf6-4a6a-b57e-cd9c6b9c2822-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2v8g7\" (UID: \"fd53a7ee-bdf6-4a6a-b57e-cd9c6b9c2822\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2v8g7" Jul 13 22:04:30.929278 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:30.929257 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgb98\" (UniqueName: \"kubernetes.io/projected/fd53a7ee-bdf6-4a6a-b57e-cd9c6b9c2822-kube-api-access-xgb98\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2v8g7\" (UID: \"fd53a7ee-bdf6-4a6a-b57e-cd9c6b9c2822\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2v8g7" Jul 13 22:04:31.034065 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:31.033973 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2v8g7" Jul 13 22:04:31.164571 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:31.164542 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2v8g7"] Jul 13 22:04:31.167540 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:04:31.167502 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd53a7ee_bdf6_4a6a_b57e_cd9c6b9c2822.slice/crio-9b4a205f93b611ee74e9e8c75e2d28810ccf51b7fae54b9ddd483e917f311e1b WatchSource:0}: Error finding container 9b4a205f93b611ee74e9e8c75e2d28810ccf51b7fae54b9ddd483e917f311e1b: Status 404 returned error can't find the container with id 9b4a205f93b611ee74e9e8c75e2d28810ccf51b7fae54b9ddd483e917f311e1b Jul 13 22:04:31.623794 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:31.623764 2555 generic.go:356] "Generic (PLEG): container finished" podID="fd53a7ee-bdf6-4a6a-b57e-cd9c6b9c2822" containerID="ae30605186efa3bdcba9f195fee4012929b3b0eac3b973226db0f698b7f6a152" exitCode=0 Jul 13 22:04:31.623978 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:31.623840 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2v8g7" event={"ID":"fd53a7ee-bdf6-4a6a-b57e-cd9c6b9c2822","Type":"ContainerDied","Data":"ae30605186efa3bdcba9f195fee4012929b3b0eac3b973226db0f698b7f6a152"} Jul 13 22:04:31.623978 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:31.623876 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2v8g7" event={"ID":"fd53a7ee-bdf6-4a6a-b57e-cd9c6b9c2822","Type":"ContainerStarted","Data":"9b4a205f93b611ee74e9e8c75e2d28810ccf51b7fae54b9ddd483e917f311e1b"} Jul 13 22:04:32.630068 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:32.630036 2555 generic.go:356] "Generic (PLEG): container finished" podID="fd53a7ee-bdf6-4a6a-b57e-cd9c6b9c2822" containerID="d59e9ca67d08370213549497783f3ce598adb9cfd9840121c878a9b37686909a" exitCode=0 Jul 13 22:04:32.630450 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:32.630078 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2v8g7" event={"ID":"fd53a7ee-bdf6-4a6a-b57e-cd9c6b9c2822","Type":"ContainerDied","Data":"d59e9ca67d08370213549497783f3ce598adb9cfd9840121c878a9b37686909a"} Jul 13 22:04:33.636226 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:33.636167 2555 generic.go:356] "Generic (PLEG): container finished" podID="fd53a7ee-bdf6-4a6a-b57e-cd9c6b9c2822" containerID="2508bdf772a4566849ecbec98fafd72b25b81045fb8a7bcb832e08846fb3fae6" exitCode=0 Jul 13 22:04:33.636572 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:33.636251 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2v8g7" event={"ID":"fd53a7ee-bdf6-4a6a-b57e-cd9c6b9c2822","Type":"ContainerDied","Data":"2508bdf772a4566849ecbec98fafd72b25b81045fb8a7bcb832e08846fb3fae6"} Jul 13 22:04:34.768328 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:34.768307 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2v8g7" Jul 13 22:04:34.951247 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:34.951121 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd53a7ee-bdf6-4a6a-b57e-cd9c6b9c2822-util\") pod \"fd53a7ee-bdf6-4a6a-b57e-cd9c6b9c2822\" (UID: \"fd53a7ee-bdf6-4a6a-b57e-cd9c6b9c2822\") " Jul 13 22:04:34.951247 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:34.951176 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xgb98\" (UniqueName: \"kubernetes.io/projected/fd53a7ee-bdf6-4a6a-b57e-cd9c6b9c2822-kube-api-access-xgb98\") pod \"fd53a7ee-bdf6-4a6a-b57e-cd9c6b9c2822\" (UID: \"fd53a7ee-bdf6-4a6a-b57e-cd9c6b9c2822\") " Jul 13 22:04:34.951247 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:34.951225 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd53a7ee-bdf6-4a6a-b57e-cd9c6b9c2822-bundle\") pod \"fd53a7ee-bdf6-4a6a-b57e-cd9c6b9c2822\" (UID: \"fd53a7ee-bdf6-4a6a-b57e-cd9c6b9c2822\") " Jul 13 22:04:34.951954 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:34.951919 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd53a7ee-bdf6-4a6a-b57e-cd9c6b9c2822-bundle" (OuterVolumeSpecName: "bundle") pod "fd53a7ee-bdf6-4a6a-b57e-cd9c6b9c2822" (UID: "fd53a7ee-bdf6-4a6a-b57e-cd9c6b9c2822"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:04:34.953389 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:34.953359 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd53a7ee-bdf6-4a6a-b57e-cd9c6b9c2822-kube-api-access-xgb98" (OuterVolumeSpecName: "kube-api-access-xgb98") pod "fd53a7ee-bdf6-4a6a-b57e-cd9c6b9c2822" (UID: "fd53a7ee-bdf6-4a6a-b57e-cd9c6b9c2822"). InnerVolumeSpecName "kube-api-access-xgb98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:04:34.956668 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:34.956647 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd53a7ee-bdf6-4a6a-b57e-cd9c6b9c2822-util" (OuterVolumeSpecName: "util") pod "fd53a7ee-bdf6-4a6a-b57e-cd9c6b9c2822" (UID: "fd53a7ee-bdf6-4a6a-b57e-cd9c6b9c2822"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:04:35.051943 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:35.051890 2555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd53a7ee-bdf6-4a6a-b57e-cd9c6b9c2822-util\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:04:35.051943 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:35.051935 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgb98\" (UniqueName: \"kubernetes.io/projected/fd53a7ee-bdf6-4a6a-b57e-cd9c6b9c2822-kube-api-access-xgb98\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:04:35.051943 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:35.051946 2555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd53a7ee-bdf6-4a6a-b57e-cd9c6b9c2822-bundle\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:04:35.645056 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:35.645018 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2v8g7" Jul 13 22:04:35.645236 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:35.645027 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2v8g7" event={"ID":"fd53a7ee-bdf6-4a6a-b57e-cd9c6b9c2822","Type":"ContainerDied","Data":"9b4a205f93b611ee74e9e8c75e2d28810ccf51b7fae54b9ddd483e917f311e1b"} Jul 13 22:04:35.645236 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:35.645136 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b4a205f93b611ee74e9e8c75e2d28810ccf51b7fae54b9ddd483e917f311e1b" Jul 13 22:04:46.057432 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:46.057399 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67ddb65fc-dwcsd"] Jul 13 22:04:46.057802 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:46.057644 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd53a7ee-bdf6-4a6a-b57e-cd9c6b9c2822" containerName="extract" Jul 13 22:04:46.057802 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:46.057654 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd53a7ee-bdf6-4a6a-b57e-cd9c6b9c2822" containerName="extract" Jul 13 22:04:46.057802 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:46.057665 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd53a7ee-bdf6-4a6a-b57e-cd9c6b9c2822" containerName="util" Jul 13 22:04:46.057802 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:46.057670 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd53a7ee-bdf6-4a6a-b57e-cd9c6b9c2822" containerName="util" Jul 13 22:04:46.057802 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:46.057679 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd53a7ee-bdf6-4a6a-b57e-cd9c6b9c2822" containerName="pull" Jul 13 22:04:46.057802 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:46.057685 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd53a7ee-bdf6-4a6a-b57e-cd9c6b9c2822" containerName="pull" Jul 13 22:04:46.057802 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:46.057733 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd53a7ee-bdf6-4a6a-b57e-cd9c6b9c2822" containerName="extract" Jul 13 22:04:46.059431 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:46.059416 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-67ddb65fc-dwcsd" Jul 13 22:04:46.063983 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:46.063962 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 13 22:04:46.064219 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:46.064190 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 13 22:04:46.064291 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:46.064254 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-kwtxt\"" Jul 13 22:04:46.065070 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:46.065050 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 13 22:04:46.065187 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:46.065084 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 13 22:04:46.086660 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:46.086638 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67ddb65fc-dwcsd"] Jul 13 22:04:46.133406 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:46.133374 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2f424140-dd4b-4331-8e78-cd10ee0f70e2-manifests\") pod \"opendatahub-operator-controller-manager-67ddb65fc-dwcsd\" (UID: \"2f424140-dd4b-4331-8e78-cd10ee0f70e2\") " pod="opendatahub/opendatahub-operator-controller-manager-67ddb65fc-dwcsd" Jul 13 22:04:46.133532 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:46.133414 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2f424140-dd4b-4331-8e78-cd10ee0f70e2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67ddb65fc-dwcsd\" (UID: \"2f424140-dd4b-4331-8e78-cd10ee0f70e2\") " pod="opendatahub/opendatahub-operator-controller-manager-67ddb65fc-dwcsd" Jul 13 22:04:46.133532 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:46.133465 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f424140-dd4b-4331-8e78-cd10ee0f70e2-tmp\") pod \"opendatahub-operator-controller-manager-67ddb65fc-dwcsd\" (UID: \"2f424140-dd4b-4331-8e78-cd10ee0f70e2\") " pod="opendatahub/opendatahub-operator-controller-manager-67ddb65fc-dwcsd" Jul 13 22:04:46.133532 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:46.133496 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2f424140-dd4b-4331-8e78-cd10ee0f70e2-webhook-cert\") pod \"opendatahub-operator-controller-manager-67ddb65fc-dwcsd\" (UID: \"2f424140-dd4b-4331-8e78-cd10ee0f70e2\") " pod="opendatahub/opendatahub-operator-controller-manager-67ddb65fc-dwcsd" Jul 13 22:04:46.133646 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:46.133557 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks7m5\" (UniqueName: \"kubernetes.io/projected/2f424140-dd4b-4331-8e78-cd10ee0f70e2-kube-api-access-ks7m5\") pod \"opendatahub-operator-controller-manager-67ddb65fc-dwcsd\" (UID: \"2f424140-dd4b-4331-8e78-cd10ee0f70e2\") " pod="opendatahub/opendatahub-operator-controller-manager-67ddb65fc-dwcsd" Jul 13 22:04:46.234829 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:46.234792 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks7m5\" (UniqueName: \"kubernetes.io/projected/2f424140-dd4b-4331-8e78-cd10ee0f70e2-kube-api-access-ks7m5\") pod \"opendatahub-operator-controller-manager-67ddb65fc-dwcsd\" (UID: \"2f424140-dd4b-4331-8e78-cd10ee0f70e2\") " pod="opendatahub/opendatahub-operator-controller-manager-67ddb65fc-dwcsd" Jul 13 22:04:46.235007 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:46.234844 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2f424140-dd4b-4331-8e78-cd10ee0f70e2-manifests\") pod \"opendatahub-operator-controller-manager-67ddb65fc-dwcsd\" (UID: \"2f424140-dd4b-4331-8e78-cd10ee0f70e2\") " pod="opendatahub/opendatahub-operator-controller-manager-67ddb65fc-dwcsd" Jul 13 22:04:46.235007 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:46.234966 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2f424140-dd4b-4331-8e78-cd10ee0f70e2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67ddb65fc-dwcsd\" (UID: \"2f424140-dd4b-4331-8e78-cd10ee0f70e2\") " pod="opendatahub/opendatahub-operator-controller-manager-67ddb65fc-dwcsd" Jul 13 22:04:46.235112 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:46.235007 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f424140-dd4b-4331-8e78-cd10ee0f70e2-tmp\") pod \"opendatahub-operator-controller-manager-67ddb65fc-dwcsd\" (UID: \"2f424140-dd4b-4331-8e78-cd10ee0f70e2\") " pod="opendatahub/opendatahub-operator-controller-manager-67ddb65fc-dwcsd" Jul 13 22:04:46.235112 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:46.235027 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2f424140-dd4b-4331-8e78-cd10ee0f70e2-webhook-cert\") pod \"opendatahub-operator-controller-manager-67ddb65fc-dwcsd\" (UID: \"2f424140-dd4b-4331-8e78-cd10ee0f70e2\") " pod="opendatahub/opendatahub-operator-controller-manager-67ddb65fc-dwcsd" Jul 13 22:04:46.235274 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:46.235256 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2f424140-dd4b-4331-8e78-cd10ee0f70e2-manifests\") pod \"opendatahub-operator-controller-manager-67ddb65fc-dwcsd\" (UID: \"2f424140-dd4b-4331-8e78-cd10ee0f70e2\") " pod="opendatahub/opendatahub-operator-controller-manager-67ddb65fc-dwcsd" Jul 13 22:04:46.237418 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:46.237397 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f424140-dd4b-4331-8e78-cd10ee0f70e2-tmp\") pod \"opendatahub-operator-controller-manager-67ddb65fc-dwcsd\" (UID: \"2f424140-dd4b-4331-8e78-cd10ee0f70e2\") " pod="opendatahub/opendatahub-operator-controller-manager-67ddb65fc-dwcsd" Jul 13 22:04:46.237601 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:46.237585 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2f424140-dd4b-4331-8e78-cd10ee0f70e2-webhook-cert\") pod \"opendatahub-operator-controller-manager-67ddb65fc-dwcsd\" (UID: \"2f424140-dd4b-4331-8e78-cd10ee0f70e2\") " pod="opendatahub/opendatahub-operator-controller-manager-67ddb65fc-dwcsd" Jul 13 22:04:46.237638 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:46.237607 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2f424140-dd4b-4331-8e78-cd10ee0f70e2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67ddb65fc-dwcsd\" (UID: \"2f424140-dd4b-4331-8e78-cd10ee0f70e2\") " pod="opendatahub/opendatahub-operator-controller-manager-67ddb65fc-dwcsd" Jul 13 22:04:46.243509 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:46.243487 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks7m5\" (UniqueName: \"kubernetes.io/projected/2f424140-dd4b-4331-8e78-cd10ee0f70e2-kube-api-access-ks7m5\") pod \"opendatahub-operator-controller-manager-67ddb65fc-dwcsd\" (UID: \"2f424140-dd4b-4331-8e78-cd10ee0f70e2\") " pod="opendatahub/opendatahub-operator-controller-manager-67ddb65fc-dwcsd" Jul 13 22:04:46.369399 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:46.369317 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-67ddb65fc-dwcsd" Jul 13 22:04:46.531876 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:46.531826 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67ddb65fc-dwcsd"] Jul 13 22:04:46.534490 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:04:46.534462 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f424140_dd4b_4331_8e78_cd10ee0f70e2.slice/crio-ebc3835c2feed93b97b2227c2e256a2a5acc076c04db73e686f71b706c27250f WatchSource:0}: Error finding container ebc3835c2feed93b97b2227c2e256a2a5acc076c04db73e686f71b706c27250f: Status 404 returned error can't find the container with id ebc3835c2feed93b97b2227c2e256a2a5acc076c04db73e686f71b706c27250f Jul 13 22:04:46.686455 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:46.686419 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67ddb65fc-dwcsd" event={"ID":"2f424140-dd4b-4331-8e78-cd10ee0f70e2","Type":"ContainerStarted","Data":"ebc3835c2feed93b97b2227c2e256a2a5acc076c04db73e686f71b706c27250f"} Jul 13 22:04:49.707789 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:49.707750 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67ddb65fc-dwcsd" event={"ID":"2f424140-dd4b-4331-8e78-cd10ee0f70e2","Type":"ContainerStarted","Data":"330e82a30dcfb81d42f86d87e0996b19255dea22a2e36c667de9f354a5d968ee"} Jul 13 22:04:50.713629 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:50.713594 2555 generic.go:356] "Generic (PLEG): container finished" podID="2f424140-dd4b-4331-8e78-cd10ee0f70e2" containerID="330e82a30dcfb81d42f86d87e0996b19255dea22a2e36c667de9f354a5d968ee" exitCode=0 Jul 13 22:04:50.713975 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:50.713662 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67ddb65fc-dwcsd" event={"ID":"2f424140-dd4b-4331-8e78-cd10ee0f70e2","Type":"ContainerDied","Data":"330e82a30dcfb81d42f86d87e0996b19255dea22a2e36c667de9f354a5d968ee"} Jul 13 22:04:51.719005 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:51.718972 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67ddb65fc-dwcsd" event={"ID":"2f424140-dd4b-4331-8e78-cd10ee0f70e2","Type":"ContainerStarted","Data":"f63efa25dbf369e51b451b5f3e5a31e20eaf5b130a80eb2c037d55b22756b3e1"} Jul 13 22:04:51.719425 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:51.719072 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-67ddb65fc-dwcsd" Jul 13 22:04:51.761386 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:51.761340 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-67ddb65fc-dwcsd" podStartSLOduration=1.4052209869999999 podStartE2EDuration="5.761324418s" totalImagesPullingTime="3.339238075s" totalInitContainerRuntime="1.016865356s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:04:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:04:51.758730633 +0000 UTC m=+297.752484937" watchObservedRunningTime="2026-07-13 22:04:51.761324418 +0000 UTC m=+297.755078711" Jul 13 22:04:54.443075 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:04:54.443045 2555 kubelet.go:1648] "Image garbage collection succeeded" Jul 13 22:05:02.725026 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:02.724992 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-67ddb65fc-dwcsd" Jul 13 22:05:05.682559 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:05.682526 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6bf5d9c687-q77l2"] Jul 13 22:05:05.686050 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:05.686032 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6bf5d9c687-q77l2" Jul 13 22:05:05.705742 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:05.705708 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 13 22:05:05.705898 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:05.705821 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 13 22:05:05.706820 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:05.706790 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 13 22:05:05.706820 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:05.706792 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-vkctd\"" Jul 13 22:05:05.706993 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:05.706832 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 13 22:05:05.774504 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:05.774476 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6bf5d9c687-q77l2"] Jul 13 22:05:05.789438 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:05.789409 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9247a6c2-1bd5-42a4-8a3a-b7c1f43bc10a-tls-certs\") pod \"kube-auth-proxy-6bf5d9c687-q77l2\" (UID: \"9247a6c2-1bd5-42a4-8a3a-b7c1f43bc10a\") " pod="openshift-ingress/kube-auth-proxy-6bf5d9c687-q77l2" Jul 13 22:05:05.789438 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:05.789441 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9247a6c2-1bd5-42a4-8a3a-b7c1f43bc10a-tmp\") pod \"kube-auth-proxy-6bf5d9c687-q77l2\" (UID: \"9247a6c2-1bd5-42a4-8a3a-b7c1f43bc10a\") " pod="openshift-ingress/kube-auth-proxy-6bf5d9c687-q77l2" Jul 13 22:05:05.789639 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:05.789497 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prhdj\" (UniqueName: \"kubernetes.io/projected/9247a6c2-1bd5-42a4-8a3a-b7c1f43bc10a-kube-api-access-prhdj\") pod \"kube-auth-proxy-6bf5d9c687-q77l2\" (UID: \"9247a6c2-1bd5-42a4-8a3a-b7c1f43bc10a\") " pod="openshift-ingress/kube-auth-proxy-6bf5d9c687-q77l2" Jul 13 22:05:05.890818 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:05.890780 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9247a6c2-1bd5-42a4-8a3a-b7c1f43bc10a-tls-certs\") pod \"kube-auth-proxy-6bf5d9c687-q77l2\" (UID: \"9247a6c2-1bd5-42a4-8a3a-b7c1f43bc10a\") " pod="openshift-ingress/kube-auth-proxy-6bf5d9c687-q77l2" Jul 13 22:05:05.890818 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:05.890814 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9247a6c2-1bd5-42a4-8a3a-b7c1f43bc10a-tmp\") pod \"kube-auth-proxy-6bf5d9c687-q77l2\" (UID: \"9247a6c2-1bd5-42a4-8a3a-b7c1f43bc10a\") " pod="openshift-ingress/kube-auth-proxy-6bf5d9c687-q77l2" Jul 13 22:05:05.891028 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:05.890954 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prhdj\" (UniqueName: \"kubernetes.io/projected/9247a6c2-1bd5-42a4-8a3a-b7c1f43bc10a-kube-api-access-prhdj\") pod \"kube-auth-proxy-6bf5d9c687-q77l2\" (UID: \"9247a6c2-1bd5-42a4-8a3a-b7c1f43bc10a\") " pod="openshift-ingress/kube-auth-proxy-6bf5d9c687-q77l2" Jul 13 22:05:05.893180 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:05.893149 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9247a6c2-1bd5-42a4-8a3a-b7c1f43bc10a-tmp\") pod \"kube-auth-proxy-6bf5d9c687-q77l2\" (UID: \"9247a6c2-1bd5-42a4-8a3a-b7c1f43bc10a\") " pod="openshift-ingress/kube-auth-proxy-6bf5d9c687-q77l2" Jul 13 22:05:05.893460 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:05.893439 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9247a6c2-1bd5-42a4-8a3a-b7c1f43bc10a-tls-certs\") pod \"kube-auth-proxy-6bf5d9c687-q77l2\" (UID: \"9247a6c2-1bd5-42a4-8a3a-b7c1f43bc10a\") " pod="openshift-ingress/kube-auth-proxy-6bf5d9c687-q77l2" Jul 13 22:05:05.924542 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:05.924508 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prhdj\" (UniqueName: \"kubernetes.io/projected/9247a6c2-1bd5-42a4-8a3a-b7c1f43bc10a-kube-api-access-prhdj\") pod \"kube-auth-proxy-6bf5d9c687-q77l2\" (UID: \"9247a6c2-1bd5-42a4-8a3a-b7c1f43bc10a\") " pod="openshift-ingress/kube-auth-proxy-6bf5d9c687-q77l2" Jul 13 22:05:05.994747 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:05.994657 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6bf5d9c687-q77l2" Jul 13 22:05:06.145516 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:06.145482 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6bf5d9c687-q77l2"] Jul 13 22:05:06.150118 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:05:06.150090 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9247a6c2_1bd5_42a4_8a3a_b7c1f43bc10a.slice/crio-e40960685664680f00b4ea2f0fc399d8f7decdb552c4cbfcdeb3f12a6fa7c675 WatchSource:0}: Error finding container e40960685664680f00b4ea2f0fc399d8f7decdb552c4cbfcdeb3f12a6fa7c675: Status 404 returned error can't find the container with id e40960685664680f00b4ea2f0fc399d8f7decdb552c4cbfcdeb3f12a6fa7c675 Jul 13 22:05:06.151972 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:06.151958 2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 13 22:05:06.314605 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:06.314522 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-rn2vl"] Jul 13 22:05:06.319085 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:06.319067 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-rn2vl" Jul 13 22:05:06.322587 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:06.322564 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 13 22:05:06.322707 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:06.322691 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-s67px\"" Jul 13 22:05:06.334671 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:06.334645 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-rn2vl"] Jul 13 22:05:06.395577 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:06.395534 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2j6g\" (UniqueName: \"kubernetes.io/projected/4035fc64-bb79-42c7-9d2c-a95f42177716-kube-api-access-q2j6g\") pod \"odh-model-controller-7998c67dc4-rn2vl\" (UID: \"4035fc64-bb79-42c7-9d2c-a95f42177716\") " pod="opendatahub/odh-model-controller-7998c67dc4-rn2vl" Jul 13 22:05:06.395740 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:06.395590 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4035fc64-bb79-42c7-9d2c-a95f42177716-cert\") pod \"odh-model-controller-7998c67dc4-rn2vl\" (UID: \"4035fc64-bb79-42c7-9d2c-a95f42177716\") " pod="opendatahub/odh-model-controller-7998c67dc4-rn2vl" Jul 13 22:05:06.496423 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:06.496386 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4035fc64-bb79-42c7-9d2c-a95f42177716-cert\") pod \"odh-model-controller-7998c67dc4-rn2vl\" (UID: \"4035fc64-bb79-42c7-9d2c-a95f42177716\") " pod="opendatahub/odh-model-controller-7998c67dc4-rn2vl" Jul 13 22:05:06.496601 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:06.496461 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2j6g\" (UniqueName: \"kubernetes.io/projected/4035fc64-bb79-42c7-9d2c-a95f42177716-kube-api-access-q2j6g\") pod \"odh-model-controller-7998c67dc4-rn2vl\" (UID: \"4035fc64-bb79-42c7-9d2c-a95f42177716\") " pod="opendatahub/odh-model-controller-7998c67dc4-rn2vl" Jul 13 22:05:06.496601 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:05:06.496547 2555 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 13 22:05:06.496670 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:05:06.496613 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4035fc64-bb79-42c7-9d2c-a95f42177716-cert podName:4035fc64-bb79-42c7-9d2c-a95f42177716 nodeName:}" failed. No retries permitted until 2026-07-13 22:05:06.996594891 +0000 UTC m=+312.990349179 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/4035fc64-bb79-42c7-9d2c-a95f42177716-cert") pod "odh-model-controller-7998c67dc4-rn2vl" (UID: "4035fc64-bb79-42c7-9d2c-a95f42177716") : secret "odh-model-controller-webhook-cert" not found Jul 13 22:05:06.549461 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:06.549431 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2j6g\" (UniqueName: \"kubernetes.io/projected/4035fc64-bb79-42c7-9d2c-a95f42177716-kube-api-access-q2j6g\") pod \"odh-model-controller-7998c67dc4-rn2vl\" (UID: \"4035fc64-bb79-42c7-9d2c-a95f42177716\") " pod="opendatahub/odh-model-controller-7998c67dc4-rn2vl" Jul 13 22:05:06.780436 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:06.780401 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6bf5d9c687-q77l2" event={"ID":"9247a6c2-1bd5-42a4-8a3a-b7c1f43bc10a","Type":"ContainerStarted","Data":"e40960685664680f00b4ea2f0fc399d8f7decdb552c4cbfcdeb3f12a6fa7c675"} Jul 13 22:05:07.000703 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:07.000669 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4035fc64-bb79-42c7-9d2c-a95f42177716-cert\") pod \"odh-model-controller-7998c67dc4-rn2vl\" (UID: \"4035fc64-bb79-42c7-9d2c-a95f42177716\") " pod="opendatahub/odh-model-controller-7998c67dc4-rn2vl" Jul 13 22:05:07.003984 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:07.003958 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4035fc64-bb79-42c7-9d2c-a95f42177716-cert\") pod \"odh-model-controller-7998c67dc4-rn2vl\" (UID: \"4035fc64-bb79-42c7-9d2c-a95f42177716\") " pod="opendatahub/odh-model-controller-7998c67dc4-rn2vl" Jul 13 22:05:07.229229 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:07.229163 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-rn2vl" Jul 13 22:05:07.402472 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:07.402436 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-rn2vl"] Jul 13 22:05:07.413283 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:05:07.413022 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4035fc64_bb79_42c7_9d2c_a95f42177716.slice/crio-f703f98a3ca8adbcc71dca54ad8e6d4b3bf7e20574412d6c6b0aef29dfa8097b WatchSource:0}: Error finding container f703f98a3ca8adbcc71dca54ad8e6d4b3bf7e20574412d6c6b0aef29dfa8097b: Status 404 returned error can't find the container with id f703f98a3ca8adbcc71dca54ad8e6d4b3bf7e20574412d6c6b0aef29dfa8097b Jul 13 22:05:07.788791 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:07.788750 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-rn2vl" event={"ID":"4035fc64-bb79-42c7-9d2c-a95f42177716","Type":"ContainerStarted","Data":"f703f98a3ca8adbcc71dca54ad8e6d4b3bf7e20574412d6c6b0aef29dfa8097b"} Jul 13 22:05:08.159488 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:08.159451 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-xmzbx"] Jul 13 22:05:08.162829 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:08.162720 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-xmzbx" Jul 13 22:05:08.166133 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:08.166100 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 13 22:05:08.166133 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:08.166114 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-rd95h\"" Jul 13 22:05:08.195193 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:08.195112 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-xmzbx"] Jul 13 22:05:08.219184 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:08.219147 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntplp\" (UniqueName: \"kubernetes.io/projected/77499f45-4453-4263-aa49-e3a9a534e134-kube-api-access-ntplp\") pod \"model-serving-api-54d89bd5fc-xmzbx\" (UID: \"77499f45-4453-4263-aa49-e3a9a534e134\") " pod="opendatahub/model-serving-api-54d89bd5fc-xmzbx" Jul 13 22:05:08.219398 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:08.219219 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77499f45-4453-4263-aa49-e3a9a534e134-tls-certs\") pod \"model-serving-api-54d89bd5fc-xmzbx\" (UID: \"77499f45-4453-4263-aa49-e3a9a534e134\") " pod="opendatahub/model-serving-api-54d89bd5fc-xmzbx" Jul 13 22:05:08.320410 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:08.320371 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntplp\" (UniqueName: \"kubernetes.io/projected/77499f45-4453-4263-aa49-e3a9a534e134-kube-api-access-ntplp\") pod \"model-serving-api-54d89bd5fc-xmzbx\" (UID: \"77499f45-4453-4263-aa49-e3a9a534e134\") " pod="opendatahub/model-serving-api-54d89bd5fc-xmzbx" Jul 13 22:05:08.320599 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:08.320439 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77499f45-4453-4263-aa49-e3a9a534e134-tls-certs\") pod \"model-serving-api-54d89bd5fc-xmzbx\" (UID: \"77499f45-4453-4263-aa49-e3a9a534e134\") " pod="opendatahub/model-serving-api-54d89bd5fc-xmzbx" Jul 13 22:05:08.320659 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:05:08.320625 2555 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 13 22:05:08.320722 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:05:08.320709 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77499f45-4453-4263-aa49-e3a9a534e134-tls-certs podName:77499f45-4453-4263-aa49-e3a9a534e134 nodeName:}" failed. No retries permitted until 2026-07-13 22:05:08.82068859 +0000 UTC m=+314.814442865 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/77499f45-4453-4263-aa49-e3a9a534e134-tls-certs") pod "model-serving-api-54d89bd5fc-xmzbx" (UID: "77499f45-4453-4263-aa49-e3a9a534e134") : secret "model-serving-api-tls" not found Jul 13 22:05:08.352125 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:08.352094 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntplp\" (UniqueName: \"kubernetes.io/projected/77499f45-4453-4263-aa49-e3a9a534e134-kube-api-access-ntplp\") pod \"model-serving-api-54d89bd5fc-xmzbx\" (UID: \"77499f45-4453-4263-aa49-e3a9a534e134\") " pod="opendatahub/model-serving-api-54d89bd5fc-xmzbx" Jul 13 22:05:08.680516 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:08.679098 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-8wqd9"] Jul 13 22:05:08.684724 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:08.683975 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-8wqd9" Jul 13 22:05:08.687336 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:08.686611 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-6wdkm\"" Jul 13 22:05:08.687336 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:08.686841 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 13 22:05:08.687336 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:08.687073 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 13 22:05:08.687708 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:08.687688 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 13 22:05:08.688096 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:08.688056 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 13 22:05:08.711278 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:08.710727 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-8wqd9"] Jul 13 22:05:08.824968 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:08.824920 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/90f08c6d-533d-4c21-b857-068d292fcd4c-cacerts\") pod \"istiod-openshift-gateway-7574459656-8wqd9\" (UID: \"90f08c6d-533d-4c21-b857-068d292fcd4c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8wqd9" Jul 13 22:05:08.825581 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:08.824982 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/90f08c6d-533d-4c21-b857-068d292fcd4c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-8wqd9\" (UID: \"90f08c6d-533d-4c21-b857-068d292fcd4c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8wqd9" Jul 13 22:05:08.825581 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:08.825016 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/90f08c6d-533d-4c21-b857-068d292fcd4c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-8wqd9\" (UID: \"90f08c6d-533d-4c21-b857-068d292fcd4c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8wqd9" Jul 13 22:05:08.825581 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:08.825125 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/90f08c6d-533d-4c21-b857-068d292fcd4c-local-certs\") pod \"istiod-openshift-gateway-7574459656-8wqd9\" (UID: \"90f08c6d-533d-4c21-b857-068d292fcd4c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8wqd9" Jul 13 22:05:08.825581 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:08.825184 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90f08c6d-533d-4c21-b857-068d292fcd4c-istio-token\") pod \"istiod-openshift-gateway-7574459656-8wqd9\" (UID: \"90f08c6d-533d-4c21-b857-068d292fcd4c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8wqd9" Jul 13 22:05:08.825581 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:08.825246 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/90f08c6d-533d-4c21-b857-068d292fcd4c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-8wqd9\" (UID: \"90f08c6d-533d-4c21-b857-068d292fcd4c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8wqd9" Jul 13 22:05:08.825581 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:08.825304 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77499f45-4453-4263-aa49-e3a9a534e134-tls-certs\") pod \"model-serving-api-54d89bd5fc-xmzbx\" (UID: \"77499f45-4453-4263-aa49-e3a9a534e134\") " pod="opendatahub/model-serving-api-54d89bd5fc-xmzbx" Jul 13 22:05:08.825581 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:08.825351 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrqcl\" (UniqueName: \"kubernetes.io/projected/90f08c6d-533d-4c21-b857-068d292fcd4c-kube-api-access-qrqcl\") pod \"istiod-openshift-gateway-7574459656-8wqd9\" (UID: \"90f08c6d-533d-4c21-b857-068d292fcd4c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8wqd9" Jul 13 22:05:08.829129 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:08.829100 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77499f45-4453-4263-aa49-e3a9a534e134-tls-certs\") pod \"model-serving-api-54d89bd5fc-xmzbx\" (UID: \"77499f45-4453-4263-aa49-e3a9a534e134\") " pod="opendatahub/model-serving-api-54d89bd5fc-xmzbx" Jul 13 22:05:08.926053 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:08.926014 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90f08c6d-533d-4c21-b857-068d292fcd4c-istio-token\") pod \"istiod-openshift-gateway-7574459656-8wqd9\" (UID: \"90f08c6d-533d-4c21-b857-068d292fcd4c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8wqd9" Jul 13 22:05:08.926232 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:08.926077 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/90f08c6d-533d-4c21-b857-068d292fcd4c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-8wqd9\" (UID: \"90f08c6d-533d-4c21-b857-068d292fcd4c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8wqd9" Jul 13 22:05:08.926232 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:08.926114 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrqcl\" (UniqueName: \"kubernetes.io/projected/90f08c6d-533d-4c21-b857-068d292fcd4c-kube-api-access-qrqcl\") pod \"istiod-openshift-gateway-7574459656-8wqd9\" (UID: \"90f08c6d-533d-4c21-b857-068d292fcd4c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8wqd9" Jul 13 22:05:08.926232 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:08.926165 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/90f08c6d-533d-4c21-b857-068d292fcd4c-cacerts\") pod \"istiod-openshift-gateway-7574459656-8wqd9\" (UID: \"90f08c6d-533d-4c21-b857-068d292fcd4c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8wqd9" Jul 13 22:05:08.926232 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:08.926196 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/90f08c6d-533d-4c21-b857-068d292fcd4c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-8wqd9\" (UID: \"90f08c6d-533d-4c21-b857-068d292fcd4c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8wqd9" Jul 13 22:05:08.926437 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:08.926238 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/90f08c6d-533d-4c21-b857-068d292fcd4c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-8wqd9\" (UID: \"90f08c6d-533d-4c21-b857-068d292fcd4c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8wqd9" Jul 13 22:05:08.926437 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:08.926272 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/90f08c6d-533d-4c21-b857-068d292fcd4c-local-certs\") pod \"istiod-openshift-gateway-7574459656-8wqd9\" (UID: \"90f08c6d-533d-4c21-b857-068d292fcd4c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8wqd9" Jul 13 22:05:08.926912 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:08.926850 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/90f08c6d-533d-4c21-b857-068d292fcd4c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-8wqd9\" (UID: \"90f08c6d-533d-4c21-b857-068d292fcd4c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8wqd9" Jul 13 22:05:08.929062 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:08.929033 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/90f08c6d-533d-4c21-b857-068d292fcd4c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-8wqd9\" (UID: \"90f08c6d-533d-4c21-b857-068d292fcd4c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8wqd9" Jul 13 22:05:08.930932 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:08.930851 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/90f08c6d-533d-4c21-b857-068d292fcd4c-cacerts\") pod \"istiod-openshift-gateway-7574459656-8wqd9\" (UID: \"90f08c6d-533d-4c21-b857-068d292fcd4c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8wqd9" Jul 13 22:05:08.931963 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:08.931928 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/90f08c6d-533d-4c21-b857-068d292fcd4c-local-certs\") pod \"istiod-openshift-gateway-7574459656-8wqd9\" (UID: \"90f08c6d-533d-4c21-b857-068d292fcd4c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8wqd9" Jul 13 22:05:08.932296 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:08.932274 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/90f08c6d-533d-4c21-b857-068d292fcd4c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-8wqd9\" (UID: \"90f08c6d-533d-4c21-b857-068d292fcd4c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8wqd9" Jul 13 22:05:08.935451 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:08.935415 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90f08c6d-533d-4c21-b857-068d292fcd4c-istio-token\") pod \"istiod-openshift-gateway-7574459656-8wqd9\" (UID: \"90f08c6d-533d-4c21-b857-068d292fcd4c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8wqd9" Jul 13 22:05:08.935699 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:08.935675 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrqcl\" (UniqueName: \"kubernetes.io/projected/90f08c6d-533d-4c21-b857-068d292fcd4c-kube-api-access-qrqcl\") pod \"istiod-openshift-gateway-7574459656-8wqd9\" (UID: \"90f08c6d-533d-4c21-b857-068d292fcd4c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8wqd9" Jul 13 22:05:08.997609 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:08.997577 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-8wqd9" Jul 13 22:05:09.078776 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:09.078558 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-xmzbx" Jul 13 22:05:10.540097 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:10.540075 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-xmzbx"] Jul 13 22:05:10.541541 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:05:10.541520 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77499f45_4453_4263_aa49_e3a9a534e134.slice/crio-1961674757d580c868fe3a507dd11ad99268d846f3bc0ba184705bb36c2c0108 WatchSource:0}: Error finding container 1961674757d580c868fe3a507dd11ad99268d846f3bc0ba184705bb36c2c0108: Status 404 returned error can't find the container with id 1961674757d580c868fe3a507dd11ad99268d846f3bc0ba184705bb36c2c0108 Jul 13 22:05:10.612746 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:05:10.612710 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90f08c6d_533d_4c21_b857_068d292fcd4c.slice/crio-ffe84600b46cd6bc0c56b8c23982cf9f4ae650b77e8f0dd617821aeb7175abb5 WatchSource:0}: Error finding container ffe84600b46cd6bc0c56b8c23982cf9f4ae650b77e8f0dd617821aeb7175abb5: Status 404 returned error can't find the container with id ffe84600b46cd6bc0c56b8c23982cf9f4ae650b77e8f0dd617821aeb7175abb5 Jul 13 22:05:10.613146 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:10.613125 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-8wqd9"] Jul 13 22:05:10.805084 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:10.805049 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-xmzbx" event={"ID":"77499f45-4453-4263-aa49-e3a9a534e134","Type":"ContainerStarted","Data":"1961674757d580c868fe3a507dd11ad99268d846f3bc0ba184705bb36c2c0108"} Jul 13 22:05:10.806525 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:10.806493 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-8wqd9" event={"ID":"90f08c6d-533d-4c21-b857-068d292fcd4c","Type":"ContainerStarted","Data":"ffe84600b46cd6bc0c56b8c23982cf9f4ae650b77e8f0dd617821aeb7175abb5"} Jul 13 22:05:10.808189 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:10.808162 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6bf5d9c687-q77l2" event={"ID":"9247a6c2-1bd5-42a4-8a3a-b7c1f43bc10a","Type":"ContainerStarted","Data":"48af620b034d05f9dcc17662d3a4164e1b5d200f6814ba7dec665426d5562e92"} Jul 13 22:05:10.809896 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:10.809863 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-rn2vl" event={"ID":"4035fc64-bb79-42c7-9d2c-a95f42177716","Type":"ContainerStarted","Data":"26686f2bd798f046c71259393acb92f25bfe9e8600395d2530f42da45d34b3d0"} Jul 13 22:05:10.809989 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:10.809915 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-rn2vl" Jul 13 22:05:10.867054 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:10.866999 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6bf5d9c687-q77l2" podStartSLOduration=1.6478750089999998 podStartE2EDuration="5.86698245s" totalImagesPullingTime="4.219107441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:05:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:05:10.856675062 +0000 UTC m=+316.850429356" watchObservedRunningTime="2026-07-13 22:05:10.86698245 +0000 UTC m=+316.860736742" Jul 13 22:05:10.911438 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:10.911389 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-rn2vl" podStartSLOduration=1.908875424 podStartE2EDuration="4.911375145s" totalImagesPullingTime="3.002499721s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:05:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:05:10.910690022 +0000 UTC m=+316.904444326" watchObservedRunningTime="2026-07-13 22:05:10.911375145 +0000 UTC m=+316.905129437" Jul 13 22:05:13.387415 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:13.387380 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:05:13.387780 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:13.387460 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:05:13.826732 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:13.826645 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-xmzbx" event={"ID":"77499f45-4453-4263-aa49-e3a9a534e134","Type":"ContainerStarted","Data":"fdc5d3b68b1f0f9f8d1e5876e319aed7346d6a3c2cdf4287fff48adfaa6f23a0"} Jul 13 22:05:13.826892 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:13.826764 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-xmzbx" Jul 13 22:05:13.828293 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:13.828264 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-8wqd9" event={"ID":"90f08c6d-533d-4c21-b857-068d292fcd4c","Type":"ContainerStarted","Data":"4727e8b195aea43630f87eeaaf1db7e9f920eaf62253e230cf73a82cc89c6bf5"} Jul 13 22:05:13.828382 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:13.828313 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-8wqd9" Jul 13 22:05:13.875172 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:13.875115 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-xmzbx" podStartSLOduration=3.037240916 podStartE2EDuration="5.875093748s" totalImagesPullingTime="2.837852832s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:05:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:05:13.871873908 +0000 UTC m=+319.865628234" watchObservedRunningTime="2026-07-13 22:05:13.875093748 +0000 UTC m=+319.868848043" Jul 13 22:05:13.953335 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:13.953267 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-8wqd9" podStartSLOduration=3.180996295 podStartE2EDuration="5.953247656s" totalImagesPullingTime="2.772251361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:05:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:05:13.951392913 +0000 UTC m=+319.945147225" watchObservedRunningTime="2026-07-13 22:05:13.953247656 +0000 UTC m=+319.947001949" Jul 13 22:05:14.833783 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:14.833741 2555 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-8wqd9 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 13 22:05:14.834121 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:14.833809 2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-8wqd9" podUID="90f08c6d-533d-4c21-b857-068d292fcd4c" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 13 22:05:14.952748 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:14.952715 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48rsjh"] Jul 13 22:05:14.956078 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:14.956057 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48rsjh" Jul 13 22:05:14.958517 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:14.958501 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-sfn8g\"" Jul 13 22:05:14.971533 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:14.971509 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48rsjh"] Jul 13 22:05:15.079550 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:15.079513 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0082143-c8bd-41a6-8632-3a816934bd8d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf48rsjh\" (UID: \"a0082143-c8bd-41a6-8632-3a816934bd8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48rsjh" Jul 13 22:05:15.079709 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:15.079564 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0082143-c8bd-41a6-8632-3a816934bd8d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf48rsjh\" (UID: \"a0082143-c8bd-41a6-8632-3a816934bd8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48rsjh" Jul 13 22:05:15.079709 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:15.079590 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0082143-c8bd-41a6-8632-3a816934bd8d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf48rsjh\" (UID: \"a0082143-c8bd-41a6-8632-3a816934bd8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48rsjh" Jul 13 22:05:15.079709 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:15.079617 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0082143-c8bd-41a6-8632-3a816934bd8d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf48rsjh\" (UID: \"a0082143-c8bd-41a6-8632-3a816934bd8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48rsjh" Jul 13 22:05:15.079709 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:15.079650 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0082143-c8bd-41a6-8632-3a816934bd8d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf48rsjh\" (UID: \"a0082143-c8bd-41a6-8632-3a816934bd8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48rsjh" Jul 13 22:05:15.079709 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:15.079666 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4pjw\" (UniqueName: \"kubernetes.io/projected/a0082143-c8bd-41a6-8632-3a816934bd8d-kube-api-access-v4pjw\") pod \"data-science-gateway-data-science-gateway-class-6576859cf48rsjh\" (UID: \"a0082143-c8bd-41a6-8632-3a816934bd8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48rsjh" Jul 13 22:05:15.079867 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:15.079715 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0082143-c8bd-41a6-8632-3a816934bd8d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf48rsjh\" (UID: \"a0082143-c8bd-41a6-8632-3a816934bd8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48rsjh" Jul 13 22:05:15.079867 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:15.079735 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0082143-c8bd-41a6-8632-3a816934bd8d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf48rsjh\" (UID: \"a0082143-c8bd-41a6-8632-3a816934bd8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48rsjh" Jul 13 22:05:15.079867 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:15.079753 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0082143-c8bd-41a6-8632-3a816934bd8d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf48rsjh\" (UID: \"a0082143-c8bd-41a6-8632-3a816934bd8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48rsjh" Jul 13 22:05:15.180450 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:15.180419 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0082143-c8bd-41a6-8632-3a816934bd8d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf48rsjh\" (UID: \"a0082143-c8bd-41a6-8632-3a816934bd8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48rsjh" Jul 13 22:05:15.180450 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:15.180452 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0082143-c8bd-41a6-8632-3a816934bd8d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf48rsjh\" (UID: \"a0082143-c8bd-41a6-8632-3a816934bd8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48rsjh" Jul 13 22:05:15.180690 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:15.180473 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0082143-c8bd-41a6-8632-3a816934bd8d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf48rsjh\" (UID: \"a0082143-c8bd-41a6-8632-3a816934bd8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48rsjh" Jul 13 22:05:15.180690 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:15.180492 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4pjw\" (UniqueName: \"kubernetes.io/projected/a0082143-c8bd-41a6-8632-3a816934bd8d-kube-api-access-v4pjw\") pod \"data-science-gateway-data-science-gateway-class-6576859cf48rsjh\" (UID: \"a0082143-c8bd-41a6-8632-3a816934bd8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48rsjh" Jul 13 22:05:15.180690 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:15.180528 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0082143-c8bd-41a6-8632-3a816934bd8d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf48rsjh\" (UID: \"a0082143-c8bd-41a6-8632-3a816934bd8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48rsjh" Jul 13 22:05:15.180690 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:15.180562 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0082143-c8bd-41a6-8632-3a816934bd8d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf48rsjh\" (UID: \"a0082143-c8bd-41a6-8632-3a816934bd8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48rsjh" Jul 13 22:05:15.180690 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:15.180592 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0082143-c8bd-41a6-8632-3a816934bd8d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf48rsjh\" (UID: \"a0082143-c8bd-41a6-8632-3a816934bd8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48rsjh" Jul 13 22:05:15.180690 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:15.180640 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0082143-c8bd-41a6-8632-3a816934bd8d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf48rsjh\" (UID: \"a0082143-c8bd-41a6-8632-3a816934bd8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48rsjh" Jul 13 22:05:15.180970 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:15.180695 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0082143-c8bd-41a6-8632-3a816934bd8d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf48rsjh\" (UID: \"a0082143-c8bd-41a6-8632-3a816934bd8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48rsjh" Jul 13 22:05:15.180970 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:15.180896 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0082143-c8bd-41a6-8632-3a816934bd8d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf48rsjh\" (UID: \"a0082143-c8bd-41a6-8632-3a816934bd8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48rsjh" Jul 13 22:05:15.180970 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:15.180953 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0082143-c8bd-41a6-8632-3a816934bd8d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf48rsjh\" (UID: \"a0082143-c8bd-41a6-8632-3a816934bd8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48rsjh" Jul 13 22:05:15.181128 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:15.181108 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0082143-c8bd-41a6-8632-3a816934bd8d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf48rsjh\" (UID: \"a0082143-c8bd-41a6-8632-3a816934bd8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48rsjh" Jul 13 22:05:15.181191 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:15.181174 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0082143-c8bd-41a6-8632-3a816934bd8d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf48rsjh\" (UID: \"a0082143-c8bd-41a6-8632-3a816934bd8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48rsjh" Jul 13 22:05:15.181297 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:15.181269 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0082143-c8bd-41a6-8632-3a816934bd8d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf48rsjh\" (UID: \"a0082143-c8bd-41a6-8632-3a816934bd8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48rsjh" Jul 13 22:05:15.183215 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:15.183184 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0082143-c8bd-41a6-8632-3a816934bd8d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf48rsjh\" (UID: \"a0082143-c8bd-41a6-8632-3a816934bd8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48rsjh" Jul 13 22:05:15.183284 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:15.183239 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0082143-c8bd-41a6-8632-3a816934bd8d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf48rsjh\" (UID: \"a0082143-c8bd-41a6-8632-3a816934bd8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48rsjh" Jul 13 22:05:15.191291 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:15.191268 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4pjw\" (UniqueName: \"kubernetes.io/projected/a0082143-c8bd-41a6-8632-3a816934bd8d-kube-api-access-v4pjw\") pod \"data-science-gateway-data-science-gateway-class-6576859cf48rsjh\" (UID: \"a0082143-c8bd-41a6-8632-3a816934bd8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48rsjh" Jul 13 22:05:15.197271 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:15.197248 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0082143-c8bd-41a6-8632-3a816934bd8d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf48rsjh\" (UID: \"a0082143-c8bd-41a6-8632-3a816934bd8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48rsjh" Jul 13 22:05:15.266217 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:15.266173 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48rsjh" Jul 13 22:05:15.419425 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:15.419388 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48rsjh"] Jul 13 22:05:15.422142 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:05:15.422116 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0082143_c8bd_41a6_8632_3a816934bd8d.slice/crio-3d09cc606198336054ded7b36e220bdfe1de5ced26a352a12b1af3a0e99a1226 WatchSource:0}: Error finding container 3d09cc606198336054ded7b36e220bdfe1de5ced26a352a12b1af3a0e99a1226: Status 404 returned error can't find the container with id 3d09cc606198336054ded7b36e220bdfe1de5ced26a352a12b1af3a0e99a1226 Jul 13 22:05:15.837785 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:15.837752 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48rsjh" event={"ID":"a0082143-c8bd-41a6-8632-3a816934bd8d","Type":"ContainerStarted","Data":"3d09cc606198336054ded7b36e220bdfe1de5ced26a352a12b1af3a0e99a1226"} Jul 13 22:05:17.835233 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:17.834743 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-8wqd9" Jul 13 22:05:18.092138 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:18.092086 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:05:18.092282 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:18.092186 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:05:18.092282 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:18.092263 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:05:18.183152 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:18.183125 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-5mxzw"] Jul 13 22:05:18.186420 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:18.186403 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5mxzw" Jul 13 22:05:18.189823 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:18.189804 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 13 22:05:18.191407 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:18.191384 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-vsdgx\"" Jul 13 22:05:18.198272 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:18.198250 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-5k88d"] Jul 13 22:05:18.201378 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:18.201361 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5k88d" Jul 13 22:05:18.208989 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:18.207801 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 13 22:05:18.208989 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:18.207836 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-t8wtx\"" Jul 13 22:05:18.209840 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:18.209817 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-5mxzw"] Jul 13 22:05:18.221901 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:18.221874 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-5k88d"] Jul 13 22:05:18.307725 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:18.307694 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a5adea24-b5bc-4fc6-bedd-9357092f0205-cert\") pod \"kserve-controller-manager-d6f4b65b8-5mxzw\" (UID: \"a5adea24-b5bc-4fc6-bedd-9357092f0205\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5mxzw" Jul 13 22:05:18.307887 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:18.307737 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dba67c81-79d8-4b45-8382-e7f9bfe110dc-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-5k88d\" (UID: \"dba67c81-79d8-4b45-8382-e7f9bfe110dc\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5k88d" Jul 13 22:05:18.307887 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:18.307765 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kvnw\" (UniqueName: \"kubernetes.io/projected/a5adea24-b5bc-4fc6-bedd-9357092f0205-kube-api-access-4kvnw\") pod \"kserve-controller-manager-d6f4b65b8-5mxzw\" (UID: \"a5adea24-b5bc-4fc6-bedd-9357092f0205\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5mxzw" Jul 13 22:05:18.307887 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:18.307819 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtbnl\" (UniqueName: \"kubernetes.io/projected/dba67c81-79d8-4b45-8382-e7f9bfe110dc-kube-api-access-jtbnl\") pod \"llmisvc-controller-manager-777c6bd7d5-5k88d\" (UID: \"dba67c81-79d8-4b45-8382-e7f9bfe110dc\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5k88d" Jul 13 22:05:18.409054 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:18.409017 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a5adea24-b5bc-4fc6-bedd-9357092f0205-cert\") pod \"kserve-controller-manager-d6f4b65b8-5mxzw\" (UID: \"a5adea24-b5bc-4fc6-bedd-9357092f0205\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5mxzw" Jul 13 22:05:18.409054 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:18.409061 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dba67c81-79d8-4b45-8382-e7f9bfe110dc-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-5k88d\" (UID: \"dba67c81-79d8-4b45-8382-e7f9bfe110dc\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5k88d" Jul 13 22:05:18.409330 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:18.409083 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kvnw\" (UniqueName: \"kubernetes.io/projected/a5adea24-b5bc-4fc6-bedd-9357092f0205-kube-api-access-4kvnw\") pod \"kserve-controller-manager-d6f4b65b8-5mxzw\" (UID: \"a5adea24-b5bc-4fc6-bedd-9357092f0205\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5mxzw" Jul 13 22:05:18.409330 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:18.409105 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtbnl\" (UniqueName: \"kubernetes.io/projected/dba67c81-79d8-4b45-8382-e7f9bfe110dc-kube-api-access-jtbnl\") pod \"llmisvc-controller-manager-777c6bd7d5-5k88d\" (UID: \"dba67c81-79d8-4b45-8382-e7f9bfe110dc\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5k88d" Jul 13 22:05:18.409330 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:05:18.409185 2555 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 13 22:05:18.409330 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:05:18.409269 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dba67c81-79d8-4b45-8382-e7f9bfe110dc-cert podName:dba67c81-79d8-4b45-8382-e7f9bfe110dc nodeName:}" failed. No retries permitted until 2026-07-13 22:05:18.909251595 +0000 UTC m=+324.903005865 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/dba67c81-79d8-4b45-8382-e7f9bfe110dc-cert") pod "llmisvc-controller-manager-777c6bd7d5-5k88d" (UID: "dba67c81-79d8-4b45-8382-e7f9bfe110dc") : secret "llmisvc-webhook-server-cert" not found Jul 13 22:05:18.411915 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:18.411885 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a5adea24-b5bc-4fc6-bedd-9357092f0205-cert\") pod \"kserve-controller-manager-d6f4b65b8-5mxzw\" (UID: \"a5adea24-b5bc-4fc6-bedd-9357092f0205\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5mxzw" Jul 13 22:05:18.422171 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:18.422121 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtbnl\" (UniqueName: \"kubernetes.io/projected/dba67c81-79d8-4b45-8382-e7f9bfe110dc-kube-api-access-jtbnl\") pod \"llmisvc-controller-manager-777c6bd7d5-5k88d\" (UID: \"dba67c81-79d8-4b45-8382-e7f9bfe110dc\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5k88d" Jul 13 22:05:18.422628 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:18.422596 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kvnw\" (UniqueName: \"kubernetes.io/projected/a5adea24-b5bc-4fc6-bedd-9357092f0205-kube-api-access-4kvnw\") pod \"kserve-controller-manager-d6f4b65b8-5mxzw\" (UID: \"a5adea24-b5bc-4fc6-bedd-9357092f0205\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5mxzw" Jul 13 22:05:18.501780 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:18.501746 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5mxzw" Jul 13 22:05:18.664213 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:18.664176 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-5mxzw"] Jul 13 22:05:18.666550 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:05:18.666523 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5adea24_b5bc_4fc6_bedd_9357092f0205.slice/crio-50c85a9a43b8bb9b405a5796cd9fd005bcf162db1f0dcb90b8cdec72e177737e WatchSource:0}: Error finding container 50c85a9a43b8bb9b405a5796cd9fd005bcf162db1f0dcb90b8cdec72e177737e: Status 404 returned error can't find the container with id 50c85a9a43b8bb9b405a5796cd9fd005bcf162db1f0dcb90b8cdec72e177737e Jul 13 22:05:18.746993 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:18.746963 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-8wqd9"] Jul 13 22:05:18.747213 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:18.747138 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-8wqd9" podUID="90f08c6d-533d-4c21-b857-068d292fcd4c" containerName="discovery" containerID="cri-o://4727e8b195aea43630f87eeaaf1db7e9f920eaf62253e230cf73a82cc89c6bf5" gracePeriod=30 Jul 13 22:05:18.853302 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:18.853270 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5mxzw" event={"ID":"a5adea24-b5bc-4fc6-bedd-9357092f0205","Type":"ContainerStarted","Data":"50c85a9a43b8bb9b405a5796cd9fd005bcf162db1f0dcb90b8cdec72e177737e"} Jul 13 22:05:18.854939 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:18.854914 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48rsjh" event={"ID":"a0082143-c8bd-41a6-8632-3a816934bd8d","Type":"ContainerStarted","Data":"911519f2745228488a5f1c942194f2b5a7d837f763c14575236f8de2b0ceb5a9"} Jul 13 22:05:18.882440 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:18.881439 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48rsjh" podStartSLOduration=2.213902509 podStartE2EDuration="4.881420649s" totalImagesPullingTime="2.66751814s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:05:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:05:18.879670152 +0000 UTC m=+324.873424466" watchObservedRunningTime="2026-07-13 22:05:18.881420649 +0000 UTC m=+324.875174943" Jul 13 22:05:18.914121 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:18.914037 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dba67c81-79d8-4b45-8382-e7f9bfe110dc-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-5k88d\" (UID: \"dba67c81-79d8-4b45-8382-e7f9bfe110dc\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5k88d" Jul 13 22:05:18.916579 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:18.916555 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dba67c81-79d8-4b45-8382-e7f9bfe110dc-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-5k88d\" (UID: \"dba67c81-79d8-4b45-8382-e7f9bfe110dc\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5k88d" Jul 13 22:05:18.982953 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:18.982925 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-8wqd9" Jul 13 22:05:19.113625 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:19.113594 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5k88d" Jul 13 22:05:19.114749 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:19.114730 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qrqcl\" (UniqueName: \"kubernetes.io/projected/90f08c6d-533d-4c21-b857-068d292fcd4c-kube-api-access-qrqcl\") pod \"90f08c6d-533d-4c21-b857-068d292fcd4c\" (UID: \"90f08c6d-533d-4c21-b857-068d292fcd4c\") " Jul 13 22:05:19.114837 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:19.114761 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/90f08c6d-533d-4c21-b857-068d292fcd4c-cacerts\") pod \"90f08c6d-533d-4c21-b857-068d292fcd4c\" (UID: \"90f08c6d-533d-4c21-b857-068d292fcd4c\") " Jul 13 22:05:19.114837 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:19.114794 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90f08c6d-533d-4c21-b857-068d292fcd4c-istio-token\") pod \"90f08c6d-533d-4c21-b857-068d292fcd4c\" (UID: \"90f08c6d-533d-4c21-b857-068d292fcd4c\") " Jul 13 22:05:19.114933 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:19.114917 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/90f08c6d-533d-4c21-b857-068d292fcd4c-istio-kubeconfig\") pod \"90f08c6d-533d-4c21-b857-068d292fcd4c\" (UID: \"90f08c6d-533d-4c21-b857-068d292fcd4c\") " Jul 13 22:05:19.114968 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:19.114956 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/90f08c6d-533d-4c21-b857-068d292fcd4c-istio-csr-dns-cert\") pod \"90f08c6d-533d-4c21-b857-068d292fcd4c\" (UID: \"90f08c6d-533d-4c21-b857-068d292fcd4c\") " Jul 13 22:05:19.115011 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:19.114992 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/90f08c6d-533d-4c21-b857-068d292fcd4c-local-certs\") pod \"90f08c6d-533d-4c21-b857-068d292fcd4c\" (UID: \"90f08c6d-533d-4c21-b857-068d292fcd4c\") " Jul 13 22:05:19.115059 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:19.115043 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/90f08c6d-533d-4c21-b857-068d292fcd4c-istio-csr-ca-configmap\") pod \"90f08c6d-533d-4c21-b857-068d292fcd4c\" (UID: \"90f08c6d-533d-4c21-b857-068d292fcd4c\") " Jul 13 22:05:19.115913 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:19.115747 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90f08c6d-533d-4c21-b857-068d292fcd4c-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "90f08c6d-533d-4c21-b857-068d292fcd4c" (UID: "90f08c6d-533d-4c21-b857-068d292fcd4c"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:05:19.117934 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:19.117821 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90f08c6d-533d-4c21-b857-068d292fcd4c-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "90f08c6d-533d-4c21-b857-068d292fcd4c" (UID: "90f08c6d-533d-4c21-b857-068d292fcd4c"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 22:05:19.118150 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:19.118027 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90f08c6d-533d-4c21-b857-068d292fcd4c-kube-api-access-qrqcl" (OuterVolumeSpecName: "kube-api-access-qrqcl") pod "90f08c6d-533d-4c21-b857-068d292fcd4c" (UID: "90f08c6d-533d-4c21-b857-068d292fcd4c"). InnerVolumeSpecName "kube-api-access-qrqcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:05:19.118150 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:19.118046 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90f08c6d-533d-4c21-b857-068d292fcd4c-cacerts" (OuterVolumeSpecName: "cacerts") pod "90f08c6d-533d-4c21-b857-068d292fcd4c" (UID: "90f08c6d-533d-4c21-b857-068d292fcd4c"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 22:05:19.118150 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:19.118053 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90f08c6d-533d-4c21-b857-068d292fcd4c-istio-token" (OuterVolumeSpecName: "istio-token") pod "90f08c6d-533d-4c21-b857-068d292fcd4c" (UID: "90f08c6d-533d-4c21-b857-068d292fcd4c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:05:19.118150 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:19.118116 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90f08c6d-533d-4c21-b857-068d292fcd4c-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "90f08c6d-533d-4c21-b857-068d292fcd4c" (UID: "90f08c6d-533d-4c21-b857-068d292fcd4c"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 22:05:19.118438 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:19.118256 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90f08c6d-533d-4c21-b857-068d292fcd4c-local-certs" (OuterVolumeSpecName: "local-certs") pod "90f08c6d-533d-4c21-b857-068d292fcd4c" (UID: "90f08c6d-533d-4c21-b857-068d292fcd4c"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:05:19.215712 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:19.215677 2555 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/90f08c6d-533d-4c21-b857-068d292fcd4c-local-certs\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:05:19.215712 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:19.215705 2555 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/90f08c6d-533d-4c21-b857-068d292fcd4c-istio-csr-ca-configmap\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:05:19.215712 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:19.215715 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrqcl\" (UniqueName: \"kubernetes.io/projected/90f08c6d-533d-4c21-b857-068d292fcd4c-kube-api-access-qrqcl\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:05:19.216004 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:19.215725 2555 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/90f08c6d-533d-4c21-b857-068d292fcd4c-cacerts\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:05:19.216004 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:19.215737 2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90f08c6d-533d-4c21-b857-068d292fcd4c-istio-token\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:05:19.216004 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:19.215748 2555 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/90f08c6d-533d-4c21-b857-068d292fcd4c-istio-kubeconfig\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:05:19.216004 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:19.215758 2555 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/90f08c6d-533d-4c21-b857-068d292fcd4c-istio-csr-dns-cert\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:05:19.248655 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:19.248633 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-5k88d"] Jul 13 22:05:19.250522 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:05:19.250497 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poddba67c81_79d8_4b45_8382_e7f9bfe110dc.slice/crio-24ad47a40a3f2f5a811b7fb967038e2a8e6c4ebcef40554653ff8ec13f1f63d1 WatchSource:0}: Error finding container 24ad47a40a3f2f5a811b7fb967038e2a8e6c4ebcef40554653ff8ec13f1f63d1: Status 404 returned error can't find the container with id 24ad47a40a3f2f5a811b7fb967038e2a8e6c4ebcef40554653ff8ec13f1f63d1 Jul 13 22:05:19.267110 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:19.267083 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48rsjh" Jul 13 22:05:19.271776 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:19.271757 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48rsjh" Jul 13 22:05:19.860761 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:19.860723 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5k88d" event={"ID":"dba67c81-79d8-4b45-8382-e7f9bfe110dc","Type":"ContainerStarted","Data":"24ad47a40a3f2f5a811b7fb967038e2a8e6c4ebcef40554653ff8ec13f1f63d1"} Jul 13 22:05:19.862256 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:19.862172 2555 generic.go:356] "Generic (PLEG): container finished" podID="90f08c6d-533d-4c21-b857-068d292fcd4c" containerID="4727e8b195aea43630f87eeaaf1db7e9f920eaf62253e230cf73a82cc89c6bf5" exitCode=0 Jul 13 22:05:19.862256 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:19.862228 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-8wqd9" event={"ID":"90f08c6d-533d-4c21-b857-068d292fcd4c","Type":"ContainerDied","Data":"4727e8b195aea43630f87eeaaf1db7e9f920eaf62253e230cf73a82cc89c6bf5"} Jul 13 22:05:19.862440 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:19.862286 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-8wqd9" event={"ID":"90f08c6d-533d-4c21-b857-068d292fcd4c","Type":"ContainerDied","Data":"ffe84600b46cd6bc0c56b8c23982cf9f4ae650b77e8f0dd617821aeb7175abb5"} Jul 13 22:05:19.862440 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:19.862328 2555 scope.go:117] "RemoveContainer" containerID="4727e8b195aea43630f87eeaaf1db7e9f920eaf62253e230cf73a82cc89c6bf5" Jul 13 22:05:19.862440 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:19.862239 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-8wqd9" Jul 13 22:05:19.862948 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:19.862784 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48rsjh" Jul 13 22:05:19.864046 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:19.863945 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf48rsjh" Jul 13 22:05:19.873730 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:19.873713 2555 scope.go:117] "RemoveContainer" containerID="4727e8b195aea43630f87eeaaf1db7e9f920eaf62253e230cf73a82cc89c6bf5" Jul 13 22:05:19.874047 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:05:19.874017 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4727e8b195aea43630f87eeaaf1db7e9f920eaf62253e230cf73a82cc89c6bf5\": container with ID starting with 4727e8b195aea43630f87eeaaf1db7e9f920eaf62253e230cf73a82cc89c6bf5 not found: ID does not exist" containerID="4727e8b195aea43630f87eeaaf1db7e9f920eaf62253e230cf73a82cc89c6bf5" Jul 13 22:05:19.874138 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:19.874058 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4727e8b195aea43630f87eeaaf1db7e9f920eaf62253e230cf73a82cc89c6bf5"} err="failed to get container status \"4727e8b195aea43630f87eeaaf1db7e9f920eaf62253e230cf73a82cc89c6bf5\": rpc error: code = NotFound desc = could not find container \"4727e8b195aea43630f87eeaaf1db7e9f920eaf62253e230cf73a82cc89c6bf5\": container with ID starting with 4727e8b195aea43630f87eeaaf1db7e9f920eaf62253e230cf73a82cc89c6bf5 not found: ID does not exist" Jul 13 22:05:19.925315 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:19.925288 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-8wqd9"] Jul 13 22:05:19.936021 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:19.935993 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-8wqd9"] Jul 13 22:05:20.562531 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:20.562503 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90f08c6d-533d-4c21-b857-068d292fcd4c" path="/var/lib/kubelet/pods/90f08c6d-533d-4c21-b857-068d292fcd4c/volumes" Jul 13 22:05:21.818821 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:21.818795 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-rn2vl" Jul 13 22:05:21.874414 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:21.874372 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5mxzw" event={"ID":"a5adea24-b5bc-4fc6-bedd-9357092f0205","Type":"ContainerStarted","Data":"29f50ddc2e147e9e12123d8b9db32dc28926dbc2659527a51d2ee3ea45e1ab80"} Jul 13 22:05:21.894544 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:21.894490 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5mxzw" podStartSLOduration=0.88376957 podStartE2EDuration="3.894470062s" totalImagesPullingTime="3.010700492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:05:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:05:21.893664913 +0000 UTC m=+327.887419225" watchObservedRunningTime="2026-07-13 22:05:21.894470062 +0000 UTC m=+327.888224356" Jul 13 22:05:22.880708 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:22.880672 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5k88d" event={"ID":"dba67c81-79d8-4b45-8382-e7f9bfe110dc","Type":"ContainerStarted","Data":"143e1145993c2ba3a4653bb429a5bbb96034de84d23681ff0ef1389787e4ab77"} Jul 13 22:05:22.881141 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:22.880811 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5k88d" Jul 13 22:05:22.881141 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:22.880858 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5mxzw" Jul 13 22:05:22.901619 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:22.901529 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5k88d" podStartSLOduration=1.511533233 podStartE2EDuration="4.901513891s" totalImagesPullingTime="3.389980658s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:05:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:05:22.899902033 +0000 UTC m=+328.893656353" watchObservedRunningTime="2026-07-13 22:05:22.901513891 +0000 UTC m=+328.895268187" Jul 13 22:05:24.836244 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:24.836194 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-xmzbx" Jul 13 22:05:53.887159 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:53.887071 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5k88d" Jul 13 22:05:53.890000 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:05:53.889977 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5mxzw" Jul 13 22:06:06.591728 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:06.591696 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6wdmq"] Jul 13 22:06:06.592114 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:06.591997 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90f08c6d-533d-4c21-b857-068d292fcd4c" containerName="discovery" Jul 13 22:06:06.592114 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:06.592009 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="90f08c6d-533d-4c21-b857-068d292fcd4c" containerName="discovery" Jul 13 22:06:06.592114 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:06.592075 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="90f08c6d-533d-4c21-b857-068d292fcd4c" containerName="discovery" Jul 13 22:06:06.599584 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:06.599556 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-6wdmq" Jul 13 22:06:06.602489 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:06.602469 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 13 22:06:06.602656 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:06.602509 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 13 22:06:06.602656 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:06.602578 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-qrckp\"" Jul 13 22:06:06.606382 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:06.606356 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6wdmq"] Jul 13 22:06:06.686459 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:06.686420 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5m6z\" (UniqueName: \"kubernetes.io/projected/4318cd29-cf75-45a3-8141-17013ee1c195-kube-api-access-r5m6z\") pod \"kuadrant-operator-catalog-6wdmq\" (UID: \"4318cd29-cf75-45a3-8141-17013ee1c195\") " pod="kuadrant-system/kuadrant-operator-catalog-6wdmq" Jul 13 22:06:06.787378 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:06.787331 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5m6z\" (UniqueName: \"kubernetes.io/projected/4318cd29-cf75-45a3-8141-17013ee1c195-kube-api-access-r5m6z\") pod \"kuadrant-operator-catalog-6wdmq\" (UID: \"4318cd29-cf75-45a3-8141-17013ee1c195\") " pod="kuadrant-system/kuadrant-operator-catalog-6wdmq" Jul 13 22:06:06.797508 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:06.797483 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5m6z\" (UniqueName: \"kubernetes.io/projected/4318cd29-cf75-45a3-8141-17013ee1c195-kube-api-access-r5m6z\") pod \"kuadrant-operator-catalog-6wdmq\" (UID: \"4318cd29-cf75-45a3-8141-17013ee1c195\") " pod="kuadrant-system/kuadrant-operator-catalog-6wdmq" Jul 13 22:06:06.911351 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:06.911315 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-6wdmq" Jul 13 22:06:06.960005 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:06.959974 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6wdmq"] Jul 13 22:06:07.051274 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:07.051249 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6wdmq"] Jul 13 22:06:07.053081 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:06:07.053052 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4318cd29_cf75_45a3_8141_17013ee1c195.slice/crio-d2aa956dfc5defdfec7ab2f4abb5fb53f2300146d891596cb20f241ac1525b99 WatchSource:0}: Error finding container d2aa956dfc5defdfec7ab2f4abb5fb53f2300146d891596cb20f241ac1525b99: Status 404 returned error can't find the container with id d2aa956dfc5defdfec7ab2f4abb5fb53f2300146d891596cb20f241ac1525b99 Jul 13 22:06:07.075171 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:07.075145 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-6wdmq" event={"ID":"4318cd29-cf75-45a3-8141-17013ee1c195","Type":"ContainerStarted","Data":"d2aa956dfc5defdfec7ab2f4abb5fb53f2300146d891596cb20f241ac1525b99"} Jul 13 22:06:07.170877 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:07.170793 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-85hq2"] Jul 13 22:06:07.174002 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:07.173984 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-85hq2" Jul 13 22:06:07.188436 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:07.188415 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-85hq2"] Jul 13 22:06:07.291917 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:07.291887 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62xk9\" (UniqueName: \"kubernetes.io/projected/c832fc59-c341-4ff1-bd9c-193450f27e73-kube-api-access-62xk9\") pod \"kuadrant-operator-catalog-85hq2\" (UID: \"c832fc59-c341-4ff1-bd9c-193450f27e73\") " pod="kuadrant-system/kuadrant-operator-catalog-85hq2" Jul 13 22:06:07.392717 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:07.392682 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62xk9\" (UniqueName: \"kubernetes.io/projected/c832fc59-c341-4ff1-bd9c-193450f27e73-kube-api-access-62xk9\") pod \"kuadrant-operator-catalog-85hq2\" (UID: \"c832fc59-c341-4ff1-bd9c-193450f27e73\") " pod="kuadrant-system/kuadrant-operator-catalog-85hq2" Jul 13 22:06:07.401815 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:07.401794 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62xk9\" (UniqueName: \"kubernetes.io/projected/c832fc59-c341-4ff1-bd9c-193450f27e73-kube-api-access-62xk9\") pod \"kuadrant-operator-catalog-85hq2\" (UID: \"c832fc59-c341-4ff1-bd9c-193450f27e73\") " pod="kuadrant-system/kuadrant-operator-catalog-85hq2" Jul 13 22:06:07.482916 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:07.482831 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-85hq2" Jul 13 22:06:07.617046 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:07.616984 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-85hq2"] Jul 13 22:06:07.619314 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:06:07.619287 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc832fc59_c341_4ff1_bd9c_193450f27e73.slice/crio-67e32897c5658d363c723aa822fcd7cb13bdf63a984988f242554639f7633f45 WatchSource:0}: Error finding container 67e32897c5658d363c723aa822fcd7cb13bdf63a984988f242554639f7633f45: Status 404 returned error can't find the container with id 67e32897c5658d363c723aa822fcd7cb13bdf63a984988f242554639f7633f45 Jul 13 22:06:08.082065 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:08.082034 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-85hq2" event={"ID":"c832fc59-c341-4ff1-bd9c-193450f27e73","Type":"ContainerStarted","Data":"67e32897c5658d363c723aa822fcd7cb13bdf63a984988f242554639f7633f45"} Jul 13 22:06:10.093527 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:10.093487 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-85hq2" event={"ID":"c832fc59-c341-4ff1-bd9c-193450f27e73","Type":"ContainerStarted","Data":"5a0a29527b0824b2ad018cf3177762765477d54fe967d34643c49d9e770adc03"} Jul 13 22:06:10.095057 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:10.095032 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-6wdmq" event={"ID":"4318cd29-cf75-45a3-8141-17013ee1c195","Type":"ContainerStarted","Data":"f69a8afaa410aa3b2e9aa9e87c60fad1e6f4adafb54ec4fbfa6a59d53194c9ba"} Jul 13 22:06:10.095186 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:10.095102 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-6wdmq" podUID="4318cd29-cf75-45a3-8141-17013ee1c195" containerName="registry-server" containerID="cri-o://f69a8afaa410aa3b2e9aa9e87c60fad1e6f4adafb54ec4fbfa6a59d53194c9ba" gracePeriod=2 Jul 13 22:06:10.114332 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:10.114289 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-85hq2" podStartSLOduration=1.5255894319999999 podStartE2EDuration="3.11427501s" totalImagesPullingTime="1.588685578s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:06:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:06:10.11260214 +0000 UTC m=+376.106356458" watchObservedRunningTime="2026-07-13 22:06:10.11427501 +0000 UTC m=+376.108029303" Jul 13 22:06:10.130184 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:10.130140 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-6wdmq" podStartSLOduration=1.9782038050000001 podStartE2EDuration="4.130126135s" totalImagesPullingTime="2.15192233s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:06:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:06:10.128870665 +0000 UTC m=+376.122624954" watchObservedRunningTime="2026-07-13 22:06:10.130126135 +0000 UTC m=+376.123880440" Jul 13 22:06:10.341411 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:10.341389 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-6wdmq" Jul 13 22:06:10.417194 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:10.417168 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r5m6z\" (UniqueName: \"kubernetes.io/projected/4318cd29-cf75-45a3-8141-17013ee1c195-kube-api-access-r5m6z\") pod \"4318cd29-cf75-45a3-8141-17013ee1c195\" (UID: \"4318cd29-cf75-45a3-8141-17013ee1c195\") " Jul 13 22:06:10.419382 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:10.419357 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4318cd29-cf75-45a3-8141-17013ee1c195-kube-api-access-r5m6z" (OuterVolumeSpecName: "kube-api-access-r5m6z") pod "4318cd29-cf75-45a3-8141-17013ee1c195" (UID: "4318cd29-cf75-45a3-8141-17013ee1c195"). InnerVolumeSpecName "kube-api-access-r5m6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:06:10.518142 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:10.518114 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5m6z\" (UniqueName: \"kubernetes.io/projected/4318cd29-cf75-45a3-8141-17013ee1c195-kube-api-access-r5m6z\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:06:11.100737 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:11.100707 2555 generic.go:356] "Generic (PLEG): container finished" podID="4318cd29-cf75-45a3-8141-17013ee1c195" containerID="f69a8afaa410aa3b2e9aa9e87c60fad1e6f4adafb54ec4fbfa6a59d53194c9ba" exitCode=0 Jul 13 22:06:11.101241 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:11.100759 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-6wdmq" Jul 13 22:06:11.101241 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:11.100793 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-6wdmq" event={"ID":"4318cd29-cf75-45a3-8141-17013ee1c195","Type":"ContainerDied","Data":"f69a8afaa410aa3b2e9aa9e87c60fad1e6f4adafb54ec4fbfa6a59d53194c9ba"} Jul 13 22:06:11.101241 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:11.100838 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-6wdmq" event={"ID":"4318cd29-cf75-45a3-8141-17013ee1c195","Type":"ContainerDied","Data":"d2aa956dfc5defdfec7ab2f4abb5fb53f2300146d891596cb20f241ac1525b99"} Jul 13 22:06:11.101241 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:11.100859 2555 scope.go:117] "RemoveContainer" containerID="f69a8afaa410aa3b2e9aa9e87c60fad1e6f4adafb54ec4fbfa6a59d53194c9ba" Jul 13 22:06:11.109682 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:11.109662 2555 scope.go:117] "RemoveContainer" containerID="f69a8afaa410aa3b2e9aa9e87c60fad1e6f4adafb54ec4fbfa6a59d53194c9ba" Jul 13 22:06:11.109929 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:06:11.109913 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f69a8afaa410aa3b2e9aa9e87c60fad1e6f4adafb54ec4fbfa6a59d53194c9ba\": container with ID starting with f69a8afaa410aa3b2e9aa9e87c60fad1e6f4adafb54ec4fbfa6a59d53194c9ba not found: ID does not exist" containerID="f69a8afaa410aa3b2e9aa9e87c60fad1e6f4adafb54ec4fbfa6a59d53194c9ba" Jul 13 22:06:11.109979 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:11.109938 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f69a8afaa410aa3b2e9aa9e87c60fad1e6f4adafb54ec4fbfa6a59d53194c9ba"} err="failed to get container status \"f69a8afaa410aa3b2e9aa9e87c60fad1e6f4adafb54ec4fbfa6a59d53194c9ba\": rpc error: code = NotFound desc = could not find container \"f69a8afaa410aa3b2e9aa9e87c60fad1e6f4adafb54ec4fbfa6a59d53194c9ba\": container with ID starting with f69a8afaa410aa3b2e9aa9e87c60fad1e6f4adafb54ec4fbfa6a59d53194c9ba not found: ID does not exist" Jul 13 22:06:11.122152 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:11.122124 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6wdmq"] Jul 13 22:06:11.126748 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:11.126724 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6wdmq"] Jul 13 22:06:12.559022 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:12.558990 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4318cd29-cf75-45a3-8141-17013ee1c195" path="/var/lib/kubelet/pods/4318cd29-cf75-45a3-8141-17013ee1c195/volumes" Jul 13 22:06:17.483911 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:17.483869 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-85hq2" Jul 13 22:06:17.484447 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:17.483938 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-85hq2" Jul 13 22:06:17.506811 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:17.506787 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-85hq2" Jul 13 22:06:18.153299 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:18.153273 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-85hq2" Jul 13 22:06:22.004637 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:22.004602 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k5bfz"] Jul 13 22:06:22.005002 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:22.004938 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4318cd29-cf75-45a3-8141-17013ee1c195" containerName="registry-server" Jul 13 22:06:22.005002 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:22.004949 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4318cd29-cf75-45a3-8141-17013ee1c195" containerName="registry-server" Jul 13 22:06:22.005074 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:22.005019 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4318cd29-cf75-45a3-8141-17013ee1c195" containerName="registry-server" Jul 13 22:06:22.007991 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:22.007974 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k5bfz" Jul 13 22:06:22.010726 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:22.010706 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-xnhp8\"" Jul 13 22:06:22.016813 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:22.016788 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k5bfz"] Jul 13 22:06:22.111715 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:22.111686 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6e63fc16-f858-4f3c-84e5-3f89699f7250-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k5bfz\" (UID: \"6e63fc16-f858-4f3c-84e5-3f89699f7250\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k5bfz" Jul 13 22:06:22.111715 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:22.111716 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6e63fc16-f858-4f3c-84e5-3f89699f7250-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k5bfz\" (UID: \"6e63fc16-f858-4f3c-84e5-3f89699f7250\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k5bfz" Jul 13 22:06:22.111895 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:22.111757 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ln94\" (UniqueName: \"kubernetes.io/projected/6e63fc16-f858-4f3c-84e5-3f89699f7250-kube-api-access-4ln94\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k5bfz\" (UID: \"6e63fc16-f858-4f3c-84e5-3f89699f7250\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k5bfz" Jul 13 22:06:22.212946 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:22.212915 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6e63fc16-f858-4f3c-84e5-3f89699f7250-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k5bfz\" (UID: \"6e63fc16-f858-4f3c-84e5-3f89699f7250\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k5bfz" Jul 13 22:06:22.212946 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:22.212951 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6e63fc16-f858-4f3c-84e5-3f89699f7250-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k5bfz\" (UID: \"6e63fc16-f858-4f3c-84e5-3f89699f7250\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k5bfz" Jul 13 22:06:22.213172 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:22.212986 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ln94\" (UniqueName: \"kubernetes.io/projected/6e63fc16-f858-4f3c-84e5-3f89699f7250-kube-api-access-4ln94\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k5bfz\" (UID: \"6e63fc16-f858-4f3c-84e5-3f89699f7250\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k5bfz" Jul 13 22:06:22.213332 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:22.213309 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6e63fc16-f858-4f3c-84e5-3f89699f7250-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k5bfz\" (UID: \"6e63fc16-f858-4f3c-84e5-3f89699f7250\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k5bfz" Jul 13 22:06:22.213474 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:22.213453 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6e63fc16-f858-4f3c-84e5-3f89699f7250-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k5bfz\" (UID: \"6e63fc16-f858-4f3c-84e5-3f89699f7250\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k5bfz" Jul 13 22:06:22.221866 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:22.221845 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ln94\" (UniqueName: \"kubernetes.io/projected/6e63fc16-f858-4f3c-84e5-3f89699f7250-kube-api-access-4ln94\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k5bfz\" (UID: \"6e63fc16-f858-4f3c-84e5-3f89699f7250\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k5bfz" Jul 13 22:06:22.317134 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:22.317067 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k5bfz" Jul 13 22:06:22.446726 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:22.446693 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k5bfz"] Jul 13 22:06:22.450468 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:06:22.450439 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e63fc16_f858_4f3c_84e5_3f89699f7250.slice/crio-6367a04a656ef3753b7657cfde6dadfd3c2c8ca9ade52869ec400159a9a05ea0 WatchSource:0}: Error finding container 6367a04a656ef3753b7657cfde6dadfd3c2c8ca9ade52869ec400159a9a05ea0: Status 404 returned error can't find the container with id 6367a04a656ef3753b7657cfde6dadfd3c2c8ca9ade52869ec400159a9a05ea0 Jul 13 22:06:22.599615 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:22.599547 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gx9k4"] Jul 13 22:06:22.602850 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:22.602834 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gx9k4" Jul 13 22:06:22.611136 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:22.611112 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gx9k4"] Jul 13 22:06:22.717059 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:22.717030 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/464555f7-8a36-4a47-8342-dfb9dbd5bae3-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gx9k4\" (UID: \"464555f7-8a36-4a47-8342-dfb9dbd5bae3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gx9k4" Jul 13 22:06:22.717059 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:22.717062 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/464555f7-8a36-4a47-8342-dfb9dbd5bae3-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gx9k4\" (UID: \"464555f7-8a36-4a47-8342-dfb9dbd5bae3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gx9k4" Jul 13 22:06:22.717304 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:22.717096 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n85jl\" (UniqueName: \"kubernetes.io/projected/464555f7-8a36-4a47-8342-dfb9dbd5bae3-kube-api-access-n85jl\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gx9k4\" (UID: \"464555f7-8a36-4a47-8342-dfb9dbd5bae3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gx9k4" Jul 13 22:06:22.818074 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:22.818030 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/464555f7-8a36-4a47-8342-dfb9dbd5bae3-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gx9k4\" (UID: \"464555f7-8a36-4a47-8342-dfb9dbd5bae3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gx9k4" Jul 13 22:06:22.818269 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:22.818083 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/464555f7-8a36-4a47-8342-dfb9dbd5bae3-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gx9k4\" (UID: \"464555f7-8a36-4a47-8342-dfb9dbd5bae3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gx9k4" Jul 13 22:06:22.818269 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:22.818137 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n85jl\" (UniqueName: \"kubernetes.io/projected/464555f7-8a36-4a47-8342-dfb9dbd5bae3-kube-api-access-n85jl\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gx9k4\" (UID: \"464555f7-8a36-4a47-8342-dfb9dbd5bae3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gx9k4" Jul 13 22:06:22.818469 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:22.818445 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/464555f7-8a36-4a47-8342-dfb9dbd5bae3-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gx9k4\" (UID: \"464555f7-8a36-4a47-8342-dfb9dbd5bae3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gx9k4" Jul 13 22:06:22.818469 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:22.818461 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/464555f7-8a36-4a47-8342-dfb9dbd5bae3-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gx9k4\" (UID: \"464555f7-8a36-4a47-8342-dfb9dbd5bae3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gx9k4" Jul 13 22:06:22.826915 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:22.826889 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n85jl\" (UniqueName: \"kubernetes.io/projected/464555f7-8a36-4a47-8342-dfb9dbd5bae3-kube-api-access-n85jl\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gx9k4\" (UID: \"464555f7-8a36-4a47-8342-dfb9dbd5bae3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gx9k4" Jul 13 22:06:22.911801 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:22.911777 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gx9k4" Jul 13 22:06:23.014909 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:23.014879 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736cmcr"] Jul 13 22:06:23.020238 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:23.020213 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736cmcr" Jul 13 22:06:23.030113 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:23.030087 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736cmcr"] Jul 13 22:06:23.050791 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:23.050762 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gx9k4"] Jul 13 22:06:23.052883 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:06:23.052859 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod464555f7_8a36_4a47_8342_dfb9dbd5bae3.slice/crio-9db80bc9693f037396e34322f391d30f9212b376f3d54e7eea64e461e31cb283 WatchSource:0}: Error finding container 9db80bc9693f037396e34322f391d30f9212b376f3d54e7eea64e461e31cb283: Status 404 returned error can't find the container with id 9db80bc9693f037396e34322f391d30f9212b376f3d54e7eea64e461e31cb283 Jul 13 22:06:23.120540 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:23.120512 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4387c748-a3eb-44ee-9016-2688537cf27f-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736cmcr\" (UID: \"4387c748-a3eb-44ee-9016-2688537cf27f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736cmcr" Jul 13 22:06:23.120660 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:23.120546 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4387c748-a3eb-44ee-9016-2688537cf27f-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736cmcr\" (UID: \"4387c748-a3eb-44ee-9016-2688537cf27f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736cmcr" Jul 13 22:06:23.120660 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:23.120626 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr66s\" (UniqueName: \"kubernetes.io/projected/4387c748-a3eb-44ee-9016-2688537cf27f-kube-api-access-jr66s\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736cmcr\" (UID: \"4387c748-a3eb-44ee-9016-2688537cf27f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736cmcr" Jul 13 22:06:23.156598 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:23.156561 2555 generic.go:356] "Generic (PLEG): container finished" podID="464555f7-8a36-4a47-8342-dfb9dbd5bae3" containerID="785513fb75cfccb153c74e6f03a478e343b5ad952bcff26d5c1a64eca765e80d" exitCode=0 Jul 13 22:06:23.156700 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:23.156644 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gx9k4" event={"ID":"464555f7-8a36-4a47-8342-dfb9dbd5bae3","Type":"ContainerDied","Data":"785513fb75cfccb153c74e6f03a478e343b5ad952bcff26d5c1a64eca765e80d"} Jul 13 22:06:23.156746 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:23.156699 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gx9k4" event={"ID":"464555f7-8a36-4a47-8342-dfb9dbd5bae3","Type":"ContainerStarted","Data":"9db80bc9693f037396e34322f391d30f9212b376f3d54e7eea64e461e31cb283"} Jul 13 22:06:23.158321 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:23.158302 2555 generic.go:356] "Generic (PLEG): container finished" podID="6e63fc16-f858-4f3c-84e5-3f89699f7250" containerID="9967da651ed46c35ceb9cc938b8aaf984cbda65aff365ada3c481c6b418416cc" exitCode=0 Jul 13 22:06:23.158426 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:23.158341 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k5bfz" event={"ID":"6e63fc16-f858-4f3c-84e5-3f89699f7250","Type":"ContainerDied","Data":"9967da651ed46c35ceb9cc938b8aaf984cbda65aff365ada3c481c6b418416cc"} Jul 13 22:06:23.158426 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:23.158378 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k5bfz" event={"ID":"6e63fc16-f858-4f3c-84e5-3f89699f7250","Type":"ContainerStarted","Data":"6367a04a656ef3753b7657cfde6dadfd3c2c8ca9ade52869ec400159a9a05ea0"} Jul 13 22:06:23.221733 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:23.221653 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr66s\" (UniqueName: \"kubernetes.io/projected/4387c748-a3eb-44ee-9016-2688537cf27f-kube-api-access-jr66s\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736cmcr\" (UID: \"4387c748-a3eb-44ee-9016-2688537cf27f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736cmcr" Jul 13 22:06:23.221870 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:23.221769 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4387c748-a3eb-44ee-9016-2688537cf27f-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736cmcr\" (UID: \"4387c748-a3eb-44ee-9016-2688537cf27f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736cmcr" Jul 13 22:06:23.221870 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:23.221795 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4387c748-a3eb-44ee-9016-2688537cf27f-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736cmcr\" (UID: \"4387c748-a3eb-44ee-9016-2688537cf27f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736cmcr" Jul 13 22:06:23.222155 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:23.222136 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4387c748-a3eb-44ee-9016-2688537cf27f-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736cmcr\" (UID: \"4387c748-a3eb-44ee-9016-2688537cf27f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736cmcr" Jul 13 22:06:23.222244 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:23.222133 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4387c748-a3eb-44ee-9016-2688537cf27f-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736cmcr\" (UID: \"4387c748-a3eb-44ee-9016-2688537cf27f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736cmcr" Jul 13 22:06:23.233572 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:23.233548 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr66s\" (UniqueName: \"kubernetes.io/projected/4387c748-a3eb-44ee-9016-2688537cf27f-kube-api-access-jr66s\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736cmcr\" (UID: \"4387c748-a3eb-44ee-9016-2688537cf27f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736cmcr" Jul 13 22:06:23.332323 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:23.332297 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736cmcr" Jul 13 22:06:23.443655 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:23.443600 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xxfwl"] Jul 13 22:06:23.448249 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:23.448227 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xxfwl" Jul 13 22:06:23.468567 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:23.468537 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xxfwl"] Jul 13 22:06:23.494162 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:23.494102 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736cmcr"] Jul 13 22:06:23.497756 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:06:23.497732 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4387c748_a3eb_44ee_9016_2688537cf27f.slice/crio-3174619bb67f4320ffd99d1d263dc9ded9fe3a02a4334d027150db731c7ac958 WatchSource:0}: Error finding container 3174619bb67f4320ffd99d1d263dc9ded9fe3a02a4334d027150db731c7ac958: Status 404 returned error can't find the container with id 3174619bb67f4320ffd99d1d263dc9ded9fe3a02a4334d027150db731c7ac958 Jul 13 22:06:23.524932 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:23.524908 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8s94\" (UniqueName: \"kubernetes.io/projected/6803f4ff-7912-492d-8540-6bc0c47650dd-kube-api-access-r8s94\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xxfwl\" (UID: \"6803f4ff-7912-492d-8540-6bc0c47650dd\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xxfwl" Jul 13 22:06:23.525052 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:23.525032 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6803f4ff-7912-492d-8540-6bc0c47650dd-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xxfwl\" (UID: \"6803f4ff-7912-492d-8540-6bc0c47650dd\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xxfwl" Jul 13 22:06:23.525120 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:23.525080 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6803f4ff-7912-492d-8540-6bc0c47650dd-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xxfwl\" (UID: \"6803f4ff-7912-492d-8540-6bc0c47650dd\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xxfwl" Jul 13 22:06:23.626132 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:23.626106 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6803f4ff-7912-492d-8540-6bc0c47650dd-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xxfwl\" (UID: \"6803f4ff-7912-492d-8540-6bc0c47650dd\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xxfwl" Jul 13 22:06:23.626279 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:23.626143 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6803f4ff-7912-492d-8540-6bc0c47650dd-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xxfwl\" (UID: \"6803f4ff-7912-492d-8540-6bc0c47650dd\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xxfwl" Jul 13 22:06:23.626279 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:23.626170 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8s94\" (UniqueName: \"kubernetes.io/projected/6803f4ff-7912-492d-8540-6bc0c47650dd-kube-api-access-r8s94\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xxfwl\" (UID: \"6803f4ff-7912-492d-8540-6bc0c47650dd\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xxfwl" Jul 13 22:06:23.626512 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:23.626486 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6803f4ff-7912-492d-8540-6bc0c47650dd-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xxfwl\" (UID: \"6803f4ff-7912-492d-8540-6bc0c47650dd\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xxfwl" Jul 13 22:06:23.626575 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:23.626505 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6803f4ff-7912-492d-8540-6bc0c47650dd-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xxfwl\" (UID: \"6803f4ff-7912-492d-8540-6bc0c47650dd\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xxfwl" Jul 13 22:06:23.636426 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:23.636403 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8s94\" (UniqueName: \"kubernetes.io/projected/6803f4ff-7912-492d-8540-6bc0c47650dd-kube-api-access-r8s94\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xxfwl\" (UID: \"6803f4ff-7912-492d-8540-6bc0c47650dd\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xxfwl" Jul 13 22:06:23.757006 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:23.756937 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xxfwl" Jul 13 22:06:23.893595 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:23.893572 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xxfwl"] Jul 13 22:06:23.895900 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:06:23.895866 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6803f4ff_7912_492d_8540_6bc0c47650dd.slice/crio-90b626b8e21023cb6d5b35f66450166d042ab2ce9f14f2f680d73da666a5524b WatchSource:0}: Error finding container 90b626b8e21023cb6d5b35f66450166d042ab2ce9f14f2f680d73da666a5524b: Status 404 returned error can't find the container with id 90b626b8e21023cb6d5b35f66450166d042ab2ce9f14f2f680d73da666a5524b Jul 13 22:06:24.166718 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:24.166689 2555 generic.go:356] "Generic (PLEG): container finished" podID="6803f4ff-7912-492d-8540-6bc0c47650dd" containerID="f84e1d740098840a2f8b75d9c84f53984bfc850e79e91a087f6e531f6f807b1a" exitCode=0 Jul 13 22:06:24.167122 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:24.166773 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xxfwl" event={"ID":"6803f4ff-7912-492d-8540-6bc0c47650dd","Type":"ContainerDied","Data":"f84e1d740098840a2f8b75d9c84f53984bfc850e79e91a087f6e531f6f807b1a"} Jul 13 22:06:24.167122 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:24.166834 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xxfwl" event={"ID":"6803f4ff-7912-492d-8540-6bc0c47650dd","Type":"ContainerStarted","Data":"90b626b8e21023cb6d5b35f66450166d042ab2ce9f14f2f680d73da666a5524b"} Jul 13 22:06:24.168591 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:24.168568 2555 generic.go:356] "Generic (PLEG): container finished" podID="4387c748-a3eb-44ee-9016-2688537cf27f" containerID="e531eab28707b7b714543bca749709f69e0c37f2807c3868ac7f264da3bcd2b8" exitCode=0 Jul 13 22:06:24.168709 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:24.168627 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736cmcr" event={"ID":"4387c748-a3eb-44ee-9016-2688537cf27f","Type":"ContainerDied","Data":"e531eab28707b7b714543bca749709f69e0c37f2807c3868ac7f264da3bcd2b8"} Jul 13 22:06:24.168709 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:24.168664 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736cmcr" event={"ID":"4387c748-a3eb-44ee-9016-2688537cf27f","Type":"ContainerStarted","Data":"3174619bb67f4320ffd99d1d263dc9ded9fe3a02a4334d027150db731c7ac958"} Jul 13 22:06:25.175497 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:25.175462 2555 generic.go:356] "Generic (PLEG): container finished" podID="6e63fc16-f858-4f3c-84e5-3f89699f7250" containerID="7dd5aa85194d6aff2290889b30aab6a33e744e6cbbad510d9681d9cb0db4b3f9" exitCode=0 Jul 13 22:06:25.175924 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:25.175546 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k5bfz" event={"ID":"6e63fc16-f858-4f3c-84e5-3f89699f7250","Type":"ContainerDied","Data":"7dd5aa85194d6aff2290889b30aab6a33e744e6cbbad510d9681d9cb0db4b3f9"} Jul 13 22:06:25.177490 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:25.177465 2555 generic.go:356] "Generic (PLEG): container finished" podID="464555f7-8a36-4a47-8342-dfb9dbd5bae3" containerID="772190f8fbbca22c8a5080d263d88aa80ba594632a31de10de635bb98f5abbeb" exitCode=0 Jul 13 22:06:25.177579 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:25.177547 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gx9k4" event={"ID":"464555f7-8a36-4a47-8342-dfb9dbd5bae3","Type":"ContainerDied","Data":"772190f8fbbca22c8a5080d263d88aa80ba594632a31de10de635bb98f5abbeb"} Jul 13 22:06:25.179447 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:25.179426 2555 generic.go:356] "Generic (PLEG): container finished" podID="6803f4ff-7912-492d-8540-6bc0c47650dd" containerID="20c2198cd22102929983fe3d95d7a8ffcd5f175e5f253d239d2081da2751fc96" exitCode=0 Jul 13 22:06:25.179527 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:25.179464 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xxfwl" event={"ID":"6803f4ff-7912-492d-8540-6bc0c47650dd","Type":"ContainerDied","Data":"20c2198cd22102929983fe3d95d7a8ffcd5f175e5f253d239d2081da2751fc96"} Jul 13 22:06:25.410424 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:06:25.410395 2555 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6803f4ff_7912_492d_8540_6bc0c47650dd.slice/crio-dda68a08591dad3dab9b8c68f3d8bd32eae0a86d2df0a57698526b200979cba7.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6803f4ff_7912_492d_8540_6bc0c47650dd.slice/crio-conmon-dda68a08591dad3dab9b8c68f3d8bd32eae0a86d2df0a57698526b200979cba7.scope\": RecentStats: unable to find data in memory cache]" Jul 13 22:06:26.187185 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:26.187153 2555 generic.go:356] "Generic (PLEG): container finished" podID="6803f4ff-7912-492d-8540-6bc0c47650dd" containerID="dda68a08591dad3dab9b8c68f3d8bd32eae0a86d2df0a57698526b200979cba7" exitCode=0 Jul 13 22:06:26.187670 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:26.187249 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xxfwl" event={"ID":"6803f4ff-7912-492d-8540-6bc0c47650dd","Type":"ContainerDied","Data":"dda68a08591dad3dab9b8c68f3d8bd32eae0a86d2df0a57698526b200979cba7"} Jul 13 22:06:26.189141 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:26.189119 2555 generic.go:356] "Generic (PLEG): container finished" podID="4387c748-a3eb-44ee-9016-2688537cf27f" containerID="3e4ad757bdddf2881df4a2fd8720954135444d1dbf8ca1b01baa09f06daefff4" exitCode=0 Jul 13 22:06:26.189288 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:26.189228 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736cmcr" event={"ID":"4387c748-a3eb-44ee-9016-2688537cf27f","Type":"ContainerDied","Data":"3e4ad757bdddf2881df4a2fd8720954135444d1dbf8ca1b01baa09f06daefff4"} Jul 13 22:06:26.191813 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:26.191788 2555 generic.go:356] "Generic (PLEG): container finished" podID="6e63fc16-f858-4f3c-84e5-3f89699f7250" containerID="1759becc62bf051a6e367097e59e1be93d860e4ba52c321adea954b64b37694a" exitCode=0 Jul 13 22:06:26.191907 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:26.191864 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k5bfz" event={"ID":"6e63fc16-f858-4f3c-84e5-3f89699f7250","Type":"ContainerDied","Data":"1759becc62bf051a6e367097e59e1be93d860e4ba52c321adea954b64b37694a"} Jul 13 22:06:26.194111 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:26.194092 2555 generic.go:356] "Generic (PLEG): container finished" podID="464555f7-8a36-4a47-8342-dfb9dbd5bae3" containerID="b9d3df7af1279113a62198f17f603c7a106f2b8955527d70b5a05eda5bff27cc" exitCode=0 Jul 13 22:06:26.194224 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:26.194152 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gx9k4" event={"ID":"464555f7-8a36-4a47-8342-dfb9dbd5bae3","Type":"ContainerDied","Data":"b9d3df7af1279113a62198f17f603c7a106f2b8955527d70b5a05eda5bff27cc"} Jul 13 22:06:27.201277 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:27.201247 2555 generic.go:356] "Generic (PLEG): container finished" podID="4387c748-a3eb-44ee-9016-2688537cf27f" containerID="0d6cf78d604ca528d5552a0a063c7c3a8e89ec47247de1180d0b445d16c7c19e" exitCode=0 Jul 13 22:06:27.201767 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:27.201368 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736cmcr" event={"ID":"4387c748-a3eb-44ee-9016-2688537cf27f","Type":"ContainerDied","Data":"0d6cf78d604ca528d5552a0a063c7c3a8e89ec47247de1180d0b445d16c7c19e"} Jul 13 22:06:27.331164 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:27.331138 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xxfwl" Jul 13 22:06:27.384931 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:27.384897 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gx9k4" Jul 13 22:06:27.387774 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:27.387756 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k5bfz" Jul 13 22:06:27.458885 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:27.458815 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/464555f7-8a36-4a47-8342-dfb9dbd5bae3-util\") pod \"464555f7-8a36-4a47-8342-dfb9dbd5bae3\" (UID: \"464555f7-8a36-4a47-8342-dfb9dbd5bae3\") " Jul 13 22:06:27.458885 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:27.458853 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4ln94\" (UniqueName: \"kubernetes.io/projected/6e63fc16-f858-4f3c-84e5-3f89699f7250-kube-api-access-4ln94\") pod \"6e63fc16-f858-4f3c-84e5-3f89699f7250\" (UID: \"6e63fc16-f858-4f3c-84e5-3f89699f7250\") " Jul 13 22:06:27.458885 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:27.458874 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6803f4ff-7912-492d-8540-6bc0c47650dd-bundle\") pod \"6803f4ff-7912-492d-8540-6bc0c47650dd\" (UID: \"6803f4ff-7912-492d-8540-6bc0c47650dd\") " Jul 13 22:06:27.459137 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:27.458897 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6803f4ff-7912-492d-8540-6bc0c47650dd-util\") pod \"6803f4ff-7912-492d-8540-6bc0c47650dd\" (UID: \"6803f4ff-7912-492d-8540-6bc0c47650dd\") " Jul 13 22:06:27.459137 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:27.458923 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n85jl\" (UniqueName: \"kubernetes.io/projected/464555f7-8a36-4a47-8342-dfb9dbd5bae3-kube-api-access-n85jl\") pod \"464555f7-8a36-4a47-8342-dfb9dbd5bae3\" (UID: \"464555f7-8a36-4a47-8342-dfb9dbd5bae3\") " Jul 13 22:06:27.459137 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:27.458950 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6e63fc16-f858-4f3c-84e5-3f89699f7250-util\") pod \"6e63fc16-f858-4f3c-84e5-3f89699f7250\" (UID: \"6e63fc16-f858-4f3c-84e5-3f89699f7250\") " Jul 13 22:06:27.459137 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:27.458978 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/464555f7-8a36-4a47-8342-dfb9dbd5bae3-bundle\") pod \"464555f7-8a36-4a47-8342-dfb9dbd5bae3\" (UID: \"464555f7-8a36-4a47-8342-dfb9dbd5bae3\") " Jul 13 22:06:27.459137 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:27.459018 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r8s94\" (UniqueName: \"kubernetes.io/projected/6803f4ff-7912-492d-8540-6bc0c47650dd-kube-api-access-r8s94\") pod \"6803f4ff-7912-492d-8540-6bc0c47650dd\" (UID: \"6803f4ff-7912-492d-8540-6bc0c47650dd\") " Jul 13 22:06:27.459137 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:27.459053 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6e63fc16-f858-4f3c-84e5-3f89699f7250-bundle\") pod \"6e63fc16-f858-4f3c-84e5-3f89699f7250\" (UID: \"6e63fc16-f858-4f3c-84e5-3f89699f7250\") " Jul 13 22:06:27.459706 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:27.459566 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6803f4ff-7912-492d-8540-6bc0c47650dd-bundle" (OuterVolumeSpecName: "bundle") pod "6803f4ff-7912-492d-8540-6bc0c47650dd" (UID: "6803f4ff-7912-492d-8540-6bc0c47650dd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:06:27.459970 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:27.459932 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e63fc16-f858-4f3c-84e5-3f89699f7250-bundle" (OuterVolumeSpecName: "bundle") pod "6e63fc16-f858-4f3c-84e5-3f89699f7250" (UID: "6e63fc16-f858-4f3c-84e5-3f89699f7250"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:06:27.460427 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:27.460375 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/464555f7-8a36-4a47-8342-dfb9dbd5bae3-bundle" (OuterVolumeSpecName: "bundle") pod "464555f7-8a36-4a47-8342-dfb9dbd5bae3" (UID: "464555f7-8a36-4a47-8342-dfb9dbd5bae3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:06:27.461380 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:27.461353 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e63fc16-f858-4f3c-84e5-3f89699f7250-kube-api-access-4ln94" (OuterVolumeSpecName: "kube-api-access-4ln94") pod "6e63fc16-f858-4f3c-84e5-3f89699f7250" (UID: "6e63fc16-f858-4f3c-84e5-3f89699f7250"). InnerVolumeSpecName "kube-api-access-4ln94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:06:27.461380 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:27.461358 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/464555f7-8a36-4a47-8342-dfb9dbd5bae3-kube-api-access-n85jl" (OuterVolumeSpecName: "kube-api-access-n85jl") pod "464555f7-8a36-4a47-8342-dfb9dbd5bae3" (UID: "464555f7-8a36-4a47-8342-dfb9dbd5bae3"). InnerVolumeSpecName "kube-api-access-n85jl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:06:27.461810 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:27.461796 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6803f4ff-7912-492d-8540-6bc0c47650dd-kube-api-access-r8s94" (OuterVolumeSpecName: "kube-api-access-r8s94") pod "6803f4ff-7912-492d-8540-6bc0c47650dd" (UID: "6803f4ff-7912-492d-8540-6bc0c47650dd"). InnerVolumeSpecName "kube-api-access-r8s94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:06:27.466414 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:27.466380 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e63fc16-f858-4f3c-84e5-3f89699f7250-util" (OuterVolumeSpecName: "util") pod "6e63fc16-f858-4f3c-84e5-3f89699f7250" (UID: "6e63fc16-f858-4f3c-84e5-3f89699f7250"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:06:27.468102 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:27.468079 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/464555f7-8a36-4a47-8342-dfb9dbd5bae3-util" (OuterVolumeSpecName: "util") pod "464555f7-8a36-4a47-8342-dfb9dbd5bae3" (UID: "464555f7-8a36-4a47-8342-dfb9dbd5bae3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:06:27.468469 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:27.468452 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6803f4ff-7912-492d-8540-6bc0c47650dd-util" (OuterVolumeSpecName: "util") pod "6803f4ff-7912-492d-8540-6bc0c47650dd" (UID: "6803f4ff-7912-492d-8540-6bc0c47650dd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:06:27.560391 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:27.560361 2555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/464555f7-8a36-4a47-8342-dfb9dbd5bae3-util\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:06:27.560391 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:27.560386 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ln94\" (UniqueName: \"kubernetes.io/projected/6e63fc16-f858-4f3c-84e5-3f89699f7250-kube-api-access-4ln94\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:06:27.560574 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:27.560400 2555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6803f4ff-7912-492d-8540-6bc0c47650dd-bundle\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:06:27.560574 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:27.560413 2555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6803f4ff-7912-492d-8540-6bc0c47650dd-util\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:06:27.560574 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:27.560421 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n85jl\" (UniqueName: \"kubernetes.io/projected/464555f7-8a36-4a47-8342-dfb9dbd5bae3-kube-api-access-n85jl\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:06:27.560574 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:27.560429 2555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6e63fc16-f858-4f3c-84e5-3f89699f7250-util\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:06:27.560574 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:27.560436 2555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/464555f7-8a36-4a47-8342-dfb9dbd5bae3-bundle\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:06:27.560574 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:27.560444 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8s94\" (UniqueName: \"kubernetes.io/projected/6803f4ff-7912-492d-8540-6bc0c47650dd-kube-api-access-r8s94\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:06:27.560574 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:27.560451 2555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6e63fc16-f858-4f3c-84e5-3f89699f7250-bundle\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:06:28.207865 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:28.207830 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gx9k4" Jul 13 22:06:28.207865 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:28.207847 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gx9k4" event={"ID":"464555f7-8a36-4a47-8342-dfb9dbd5bae3","Type":"ContainerDied","Data":"9db80bc9693f037396e34322f391d30f9212b376f3d54e7eea64e461e31cb283"} Jul 13 22:06:28.208343 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:28.207898 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9db80bc9693f037396e34322f391d30f9212b376f3d54e7eea64e461e31cb283" Jul 13 22:06:28.209685 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:28.209658 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xxfwl" Jul 13 22:06:28.209685 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:28.209676 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xxfwl" event={"ID":"6803f4ff-7912-492d-8540-6bc0c47650dd","Type":"ContainerDied","Data":"90b626b8e21023cb6d5b35f66450166d042ab2ce9f14f2f680d73da666a5524b"} Jul 13 22:06:28.209884 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:28.209714 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90b626b8e21023cb6d5b35f66450166d042ab2ce9f14f2f680d73da666a5524b" Jul 13 22:06:28.211606 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:28.211589 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k5bfz" Jul 13 22:06:28.211707 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:28.211621 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k5bfz" event={"ID":"6e63fc16-f858-4f3c-84e5-3f89699f7250","Type":"ContainerDied","Data":"6367a04a656ef3753b7657cfde6dadfd3c2c8ca9ade52869ec400159a9a05ea0"} Jul 13 22:06:28.211707 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:28.211656 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6367a04a656ef3753b7657cfde6dadfd3c2c8ca9ade52869ec400159a9a05ea0" Jul 13 22:06:28.345083 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:28.345061 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736cmcr" Jul 13 22:06:28.468585 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:28.468505 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4387c748-a3eb-44ee-9016-2688537cf27f-bundle\") pod \"4387c748-a3eb-44ee-9016-2688537cf27f\" (UID: \"4387c748-a3eb-44ee-9016-2688537cf27f\") " Jul 13 22:06:28.468585 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:28.468557 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jr66s\" (UniqueName: \"kubernetes.io/projected/4387c748-a3eb-44ee-9016-2688537cf27f-kube-api-access-jr66s\") pod \"4387c748-a3eb-44ee-9016-2688537cf27f\" (UID: \"4387c748-a3eb-44ee-9016-2688537cf27f\") " Jul 13 22:06:28.468754 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:28.468616 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4387c748-a3eb-44ee-9016-2688537cf27f-util\") pod \"4387c748-a3eb-44ee-9016-2688537cf27f\" (UID: \"4387c748-a3eb-44ee-9016-2688537cf27f\") " Jul 13 22:06:28.469269 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:28.469238 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4387c748-a3eb-44ee-9016-2688537cf27f-bundle" (OuterVolumeSpecName: "bundle") pod "4387c748-a3eb-44ee-9016-2688537cf27f" (UID: "4387c748-a3eb-44ee-9016-2688537cf27f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:06:28.470739 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:28.470711 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4387c748-a3eb-44ee-9016-2688537cf27f-kube-api-access-jr66s" (OuterVolumeSpecName: "kube-api-access-jr66s") pod "4387c748-a3eb-44ee-9016-2688537cf27f" (UID: "4387c748-a3eb-44ee-9016-2688537cf27f"). InnerVolumeSpecName "kube-api-access-jr66s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:06:28.474946 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:28.474916 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4387c748-a3eb-44ee-9016-2688537cf27f-util" (OuterVolumeSpecName: "util") pod "4387c748-a3eb-44ee-9016-2688537cf27f" (UID: "4387c748-a3eb-44ee-9016-2688537cf27f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:06:28.569771 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:28.569748 2555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4387c748-a3eb-44ee-9016-2688537cf27f-util\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:06:28.569771 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:28.569772 2555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4387c748-a3eb-44ee-9016-2688537cf27f-bundle\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:06:28.569911 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:28.569782 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jr66s\" (UniqueName: \"kubernetes.io/projected/4387c748-a3eb-44ee-9016-2688537cf27f-kube-api-access-jr66s\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:06:29.217849 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:29.217820 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736cmcr" Jul 13 22:06:29.217849 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:29.217830 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736cmcr" event={"ID":"4387c748-a3eb-44ee-9016-2688537cf27f","Type":"ContainerDied","Data":"3174619bb67f4320ffd99d1d263dc9ded9fe3a02a4334d027150db731c7ac958"} Jul 13 22:06:29.218281 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:29.217870 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3174619bb67f4320ffd99d1d263dc9ded9fe3a02a4334d027150db731c7ac958" Jul 13 22:06:38.391826 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:38.391788 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-kxv55"] Jul 13 22:06:38.392271 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:38.392123 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4387c748-a3eb-44ee-9016-2688537cf27f" containerName="extract" Jul 13 22:06:38.392271 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:38.392135 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4387c748-a3eb-44ee-9016-2688537cf27f" containerName="extract" Jul 13 22:06:38.392271 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:38.392143 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4387c748-a3eb-44ee-9016-2688537cf27f" containerName="pull" Jul 13 22:06:38.392271 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:38.392149 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4387c748-a3eb-44ee-9016-2688537cf27f" containerName="pull" Jul 13 22:06:38.392271 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:38.392159 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e63fc16-f858-4f3c-84e5-3f89699f7250" containerName="extract" Jul 13 22:06:38.392271 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:38.392165 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e63fc16-f858-4f3c-84e5-3f89699f7250" containerName="extract" Jul 13 22:06:38.392271 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:38.392171 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6803f4ff-7912-492d-8540-6bc0c47650dd" containerName="util" Jul 13 22:06:38.392271 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:38.392176 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6803f4ff-7912-492d-8540-6bc0c47650dd" containerName="util" Jul 13 22:06:38.392271 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:38.392192 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="464555f7-8a36-4a47-8342-dfb9dbd5bae3" containerName="pull" Jul 13 22:06:38.392271 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:38.392214 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="464555f7-8a36-4a47-8342-dfb9dbd5bae3" containerName="pull" Jul 13 22:06:38.392271 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:38.392222 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e63fc16-f858-4f3c-84e5-3f89699f7250" containerName="util" Jul 13 22:06:38.392271 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:38.392227 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e63fc16-f858-4f3c-84e5-3f89699f7250" containerName="util" Jul 13 22:06:38.392271 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:38.392231 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e63fc16-f858-4f3c-84e5-3f89699f7250" containerName="pull" Jul 13 22:06:38.392271 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:38.392236 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e63fc16-f858-4f3c-84e5-3f89699f7250" containerName="pull" Jul 13 22:06:38.392271 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:38.392243 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="464555f7-8a36-4a47-8342-dfb9dbd5bae3" containerName="util" Jul 13 22:06:38.392271 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:38.392248 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="464555f7-8a36-4a47-8342-dfb9dbd5bae3" containerName="util" Jul 13 22:06:38.392271 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:38.392275 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4387c748-a3eb-44ee-9016-2688537cf27f" containerName="util" Jul 13 22:06:38.392776 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:38.392285 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4387c748-a3eb-44ee-9016-2688537cf27f" containerName="util" Jul 13 22:06:38.392776 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:38.392291 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6803f4ff-7912-492d-8540-6bc0c47650dd" containerName="pull" Jul 13 22:06:38.392776 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:38.392298 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6803f4ff-7912-492d-8540-6bc0c47650dd" containerName="pull" Jul 13 22:06:38.392776 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:38.392303 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6803f4ff-7912-492d-8540-6bc0c47650dd" containerName="extract" Jul 13 22:06:38.392776 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:38.392308 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6803f4ff-7912-492d-8540-6bc0c47650dd" containerName="extract" Jul 13 22:06:38.392776 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:38.392318 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="464555f7-8a36-4a47-8342-dfb9dbd5bae3" containerName="extract" Jul 13 22:06:38.392776 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:38.392323 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="464555f7-8a36-4a47-8342-dfb9dbd5bae3" containerName="extract" Jul 13 22:06:38.392776 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:38.392384 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4387c748-a3eb-44ee-9016-2688537cf27f" containerName="extract" Jul 13 22:06:38.392776 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:38.392393 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e63fc16-f858-4f3c-84e5-3f89699f7250" containerName="extract" Jul 13 22:06:38.392776 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:38.392399 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="464555f7-8a36-4a47-8342-dfb9dbd5bae3" containerName="extract" Jul 13 22:06:38.392776 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:38.392405 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6803f4ff-7912-492d-8540-6bc0c47650dd" containerName="extract" Jul 13 22:06:38.395668 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:38.395652 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-kxv55" Jul 13 22:06:38.398457 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:38.398436 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 13 22:06:38.398694 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:38.398677 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-4njm6\"" Jul 13 22:06:38.406894 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:38.406872 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-kxv55"] Jul 13 22:06:38.556963 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:38.556933 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpb89\" (UniqueName: \"kubernetes.io/projected/90b4731a-09ca-4b22-b625-65e0551117da-kube-api-access-wpb89\") pod \"dns-operator-controller-manager-864dd78c4c-kxv55\" (UID: \"90b4731a-09ca-4b22-b625-65e0551117da\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-kxv55" Jul 13 22:06:38.658402 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:38.658317 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpb89\" (UniqueName: \"kubernetes.io/projected/90b4731a-09ca-4b22-b625-65e0551117da-kube-api-access-wpb89\") pod \"dns-operator-controller-manager-864dd78c4c-kxv55\" (UID: \"90b4731a-09ca-4b22-b625-65e0551117da\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-kxv55" Jul 13 22:06:38.668565 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:38.668533 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpb89\" (UniqueName: \"kubernetes.io/projected/90b4731a-09ca-4b22-b625-65e0551117da-kube-api-access-wpb89\") pod \"dns-operator-controller-manager-864dd78c4c-kxv55\" (UID: \"90b4731a-09ca-4b22-b625-65e0551117da\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-kxv55" Jul 13 22:06:38.705220 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:38.705173 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-kxv55" Jul 13 22:06:38.862542 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:38.862520 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-kxv55"] Jul 13 22:06:38.864502 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:06:38.864475 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90b4731a_09ca_4b22_b625_65e0551117da.slice/crio-11ee82606a0ee9c4fd5a8b79d8730377ea5107da8922e3bde8e4333d610c6d3a WatchSource:0}: Error finding container 11ee82606a0ee9c4fd5a8b79d8730377ea5107da8922e3bde8e4333d610c6d3a: Status 404 returned error can't find the container with id 11ee82606a0ee9c4fd5a8b79d8730377ea5107da8922e3bde8e4333d610c6d3a Jul 13 22:06:39.266722 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:39.266689 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-kxv55" event={"ID":"90b4731a-09ca-4b22-b625-65e0551117da","Type":"ContainerStarted","Data":"11ee82606a0ee9c4fd5a8b79d8730377ea5107da8922e3bde8e4333d610c6d3a"} Jul 13 22:06:41.277774 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:41.277681 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-kxv55" event={"ID":"90b4731a-09ca-4b22-b625-65e0551117da","Type":"ContainerStarted","Data":"b69f2d8e04a04a9124d44db1b004d5ee1a09ad3d9668925d2114dfe5f6ceb10f"} Jul 13 22:06:41.278196 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:41.277775 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-kxv55" Jul 13 22:06:41.315734 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:41.315683 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-kxv55" podStartSLOduration=1.278814113 podStartE2EDuration="3.315667866s" totalImagesPullingTime="2.036853753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:06:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:06:41.31494397 +0000 UTC m=+407.308698263" watchObservedRunningTime="2026-07-13 22:06:41.315667866 +0000 UTC m=+407.309422158" Jul 13 22:06:41.724299 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:41.724270 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9fbr8"] Jul 13 22:06:41.728025 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:41.728009 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9fbr8" Jul 13 22:06:41.731162 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:41.731144 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-lw2cj\"" Jul 13 22:06:41.744073 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:41.744053 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9fbr8"] Jul 13 22:06:41.884580 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:41.884542 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k65d2\" (UniqueName: \"kubernetes.io/projected/51845634-9547-44e3-acdc-030c90cad1cc-kube-api-access-k65d2\") pod \"limitador-operator-controller-manager-859d7b78bd-9fbr8\" (UID: \"51845634-9547-44e3-acdc-030c90cad1cc\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9fbr8" Jul 13 22:06:41.985150 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:41.985054 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k65d2\" (UniqueName: \"kubernetes.io/projected/51845634-9547-44e3-acdc-030c90cad1cc-kube-api-access-k65d2\") pod \"limitador-operator-controller-manager-859d7b78bd-9fbr8\" (UID: \"51845634-9547-44e3-acdc-030c90cad1cc\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9fbr8" Jul 13 22:06:42.002965 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:42.002943 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k65d2\" (UniqueName: \"kubernetes.io/projected/51845634-9547-44e3-acdc-030c90cad1cc-kube-api-access-k65d2\") pod \"limitador-operator-controller-manager-859d7b78bd-9fbr8\" (UID: \"51845634-9547-44e3-acdc-030c90cad1cc\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9fbr8" Jul 13 22:06:42.037602 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:42.037578 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9fbr8" Jul 13 22:06:42.185425 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:42.185395 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9fbr8"] Jul 13 22:06:42.189004 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:06:42.188983 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51845634_9547_44e3_acdc_030c90cad1cc.slice/crio-b3c6e0fddcfd64d68b96a9c1def8e5391318560461deb1d423a5c6a971e6316b WatchSource:0}: Error finding container b3c6e0fddcfd64d68b96a9c1def8e5391318560461deb1d423a5c6a971e6316b: Status 404 returned error can't find the container with id b3c6e0fddcfd64d68b96a9c1def8e5391318560461deb1d423a5c6a971e6316b Jul 13 22:06:42.283927 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:42.283839 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9fbr8" event={"ID":"51845634-9547-44e3-acdc-030c90cad1cc","Type":"ContainerStarted","Data":"b3c6e0fddcfd64d68b96a9c1def8e5391318560461deb1d423a5c6a971e6316b"} Jul 13 22:06:44.295701 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:44.295658 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9fbr8" event={"ID":"51845634-9547-44e3-acdc-030c90cad1cc","Type":"ContainerStarted","Data":"896e47dc67871e3a75aaee619c26a197531e50fbc6ce4d05be16e59c2bd190ff"} Jul 13 22:06:44.296123 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:44.295779 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9fbr8" Jul 13 22:06:44.317981 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:44.317934 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9fbr8" podStartSLOduration=1.372732142 podStartE2EDuration="3.317918522s" totalImagesPullingTime="1.94518638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:06:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:06:44.316990774 +0000 UTC m=+410.310745093" watchObservedRunningTime="2026-07-13 22:06:44.317918522 +0000 UTC m=+410.311672815" Jul 13 22:06:46.767581 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:46.767547 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xc68s"] Jul 13 22:06:46.770682 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:46.770665 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xc68s" Jul 13 22:06:46.778807 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:46.778790 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-9jb6z\"" Jul 13 22:06:46.815323 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:46.815287 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xc68s"] Jul 13 22:06:46.823976 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:46.823813 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcr5s\" (UniqueName: \"kubernetes.io/projected/716ddd13-6b20-409e-aa1e-f0ea2d4cbdd5-kube-api-access-jcr5s\") pod \"kuadrant-operator-controller-manager-8684c9ff49-xc68s\" (UID: \"716ddd13-6b20-409e-aa1e-f0ea2d4cbdd5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xc68s" Jul 13 22:06:46.823976 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:46.823927 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/716ddd13-6b20-409e-aa1e-f0ea2d4cbdd5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-xc68s\" (UID: \"716ddd13-6b20-409e-aa1e-f0ea2d4cbdd5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xc68s" Jul 13 22:06:46.924480 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:46.924452 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/716ddd13-6b20-409e-aa1e-f0ea2d4cbdd5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-xc68s\" (UID: \"716ddd13-6b20-409e-aa1e-f0ea2d4cbdd5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xc68s" Jul 13 22:06:46.924635 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:46.924505 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcr5s\" (UniqueName: \"kubernetes.io/projected/716ddd13-6b20-409e-aa1e-f0ea2d4cbdd5-kube-api-access-jcr5s\") pod \"kuadrant-operator-controller-manager-8684c9ff49-xc68s\" (UID: \"716ddd13-6b20-409e-aa1e-f0ea2d4cbdd5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xc68s" Jul 13 22:06:46.924918 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:46.924895 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/716ddd13-6b20-409e-aa1e-f0ea2d4cbdd5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-xc68s\" (UID: \"716ddd13-6b20-409e-aa1e-f0ea2d4cbdd5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xc68s" Jul 13 22:06:46.938469 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:46.938441 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcr5s\" (UniqueName: \"kubernetes.io/projected/716ddd13-6b20-409e-aa1e-f0ea2d4cbdd5-kube-api-access-jcr5s\") pod \"kuadrant-operator-controller-manager-8684c9ff49-xc68s\" (UID: \"716ddd13-6b20-409e-aa1e-f0ea2d4cbdd5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xc68s" Jul 13 22:06:47.080568 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:47.080480 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xc68s" Jul 13 22:06:47.228508 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:47.228484 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xc68s"] Jul 13 22:06:47.230272 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:06:47.230244 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod716ddd13_6b20_409e_aa1e_f0ea2d4cbdd5.slice/crio-4559666f2db7b68249526237d4209dea9e6d4501f61692bb772cb1daf120c004 WatchSource:0}: Error finding container 4559666f2db7b68249526237d4209dea9e6d4501f61692bb772cb1daf120c004: Status 404 returned error can't find the container with id 4559666f2db7b68249526237d4209dea9e6d4501f61692bb772cb1daf120c004 Jul 13 22:06:47.312558 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:47.312524 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xc68s" event={"ID":"716ddd13-6b20-409e-aa1e-f0ea2d4cbdd5","Type":"ContainerStarted","Data":"4559666f2db7b68249526237d4209dea9e6d4501f61692bb772cb1daf120c004"} Jul 13 22:06:49.938641 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:49.938608 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-q5jsr"] Jul 13 22:06:49.942314 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:49.942291 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-q5jsr" Jul 13 22:06:49.944952 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:49.944934 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-89l7m\"" Jul 13 22:06:49.948191 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:49.948170 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqjwg\" (UniqueName: \"kubernetes.io/projected/ed25e2c9-f71c-4537-a748-081bbb48b8fd-kube-api-access-jqjwg\") pod \"authorino-operator-75cc8d5cb7-q5jsr\" (UID: \"ed25e2c9-f71c-4537-a748-081bbb48b8fd\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-q5jsr" Jul 13 22:06:49.956387 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:49.956364 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-q5jsr"] Jul 13 22:06:50.048559 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:50.048526 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqjwg\" (UniqueName: \"kubernetes.io/projected/ed25e2c9-f71c-4537-a748-081bbb48b8fd-kube-api-access-jqjwg\") pod \"authorino-operator-75cc8d5cb7-q5jsr\" (UID: \"ed25e2c9-f71c-4537-a748-081bbb48b8fd\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-q5jsr" Jul 13 22:06:50.062724 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:50.062687 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqjwg\" (UniqueName: \"kubernetes.io/projected/ed25e2c9-f71c-4537-a748-081bbb48b8fd-kube-api-access-jqjwg\") pod \"authorino-operator-75cc8d5cb7-q5jsr\" (UID: \"ed25e2c9-f71c-4537-a748-081bbb48b8fd\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-q5jsr" Jul 13 22:06:50.255650 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:50.255564 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-q5jsr" Jul 13 22:06:50.432619 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:50.432594 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-q5jsr"] Jul 13 22:06:50.434157 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:06:50.434132 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded25e2c9_f71c_4537_a748_081bbb48b8fd.slice/crio-cf2275ac778d5c1aede97b0f704ccca5063aa9913822313f270db436d633a601 WatchSource:0}: Error finding container cf2275ac778d5c1aede97b0f704ccca5063aa9913822313f270db436d633a601: Status 404 returned error can't find the container with id cf2275ac778d5c1aede97b0f704ccca5063aa9913822313f270db436d633a601 Jul 13 22:06:51.339539 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:51.339493 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-q5jsr" event={"ID":"ed25e2c9-f71c-4537-a748-081bbb48b8fd","Type":"ContainerStarted","Data":"cf2275ac778d5c1aede97b0f704ccca5063aa9913822313f270db436d633a601"} Jul 13 22:06:52.286750 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:52.286716 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-kxv55" Jul 13 22:06:53.353502 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:53.353421 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-q5jsr" event={"ID":"ed25e2c9-f71c-4537-a748-081bbb48b8fd","Type":"ContainerStarted","Data":"31f370724d0fa41a50e6a6ec1ac392904c1007d1e674751bec08c30a1c6337c9"} Jul 13 22:06:53.353502 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:53.353489 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-q5jsr" Jul 13 22:06:53.382007 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:53.381966 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-q5jsr" podStartSLOduration=1.891306374 podStartE2EDuration="4.381951462s" totalImagesPullingTime="2.490645088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:06:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:06:53.380379404 +0000 UTC m=+419.374133698" watchObservedRunningTime="2026-07-13 22:06:53.381951462 +0000 UTC m=+419.375705824" Jul 13 22:06:55.302278 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:55.302245 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9fbr8" Jul 13 22:06:55.366717 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:55.366679 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xc68s" event={"ID":"716ddd13-6b20-409e-aa1e-f0ea2d4cbdd5","Type":"ContainerStarted","Data":"8d472b2417f41e4e23a4414477b9764faeb32cfa9e973166aff6d24c4c4d47e9"} Jul 13 22:06:55.366975 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:55.366832 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xc68s" Jul 13 22:06:55.389893 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:06:55.389845 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xc68s" podStartSLOduration=1.892973283 podStartE2EDuration="9.389830999s" totalImagesPullingTime="7.496857716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:06:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:06:55.387562728 +0000 UTC m=+421.381317018" watchObservedRunningTime="2026-07-13 22:06:55.389830999 +0000 UTC m=+421.383585359" Jul 13 22:07:04.362758 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:04.362728 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-q5jsr" Jul 13 22:07:06.374003 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:06.373968 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xc68s" Jul 13 22:07:08.173408 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:08.173374 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xc68s"] Jul 13 22:07:08.173788 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:08.173559 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xc68s" podUID="716ddd13-6b20-409e-aa1e-f0ea2d4cbdd5" containerName="manager" containerID="cri-o://8d472b2417f41e4e23a4414477b9764faeb32cfa9e973166aff6d24c4c4d47e9" gracePeriod=2 Jul 13 22:07:08.186829 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:08.186766 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xc68s"] Jul 13 22:07:08.207361 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:08.207332 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9fbr8"] Jul 13 22:07:08.207560 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:08.207537 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9fbr8" podUID="51845634-9547-44e3-acdc-030c90cad1cc" containerName="manager" containerID="cri-o://896e47dc67871e3a75aaee619c26a197531e50fbc6ce4d05be16e59c2bd190ff" gracePeriod=2 Jul 13 22:07:08.210494 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:07:08.210444 2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-xc68s\" is forbidden: User \"system:node:ip-10-0-133-234.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-234.ec2.internal' and this object" podUID="716ddd13-6b20-409e-aa1e-f0ea2d4cbdd5" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xc68s" Jul 13 22:07:08.229148 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:08.229122 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9fbr8"] Jul 13 22:07:08.234076 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:08.234051 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s7x6w"] Jul 13 22:07:08.234563 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:08.234547 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51845634-9547-44e3-acdc-030c90cad1cc" containerName="manager" Jul 13 22:07:08.234610 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:08.234567 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="51845634-9547-44e3-acdc-030c90cad1cc" containerName="manager" Jul 13 22:07:08.234610 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:08.234597 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="716ddd13-6b20-409e-aa1e-f0ea2d4cbdd5" containerName="manager" Jul 13 22:07:08.234669 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:08.234608 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="716ddd13-6b20-409e-aa1e-f0ea2d4cbdd5" containerName="manager" Jul 13 22:07:08.234700 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:08.234687 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="51845634-9547-44e3-acdc-030c90cad1cc" containerName="manager" Jul 13 22:07:08.234742 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:08.234702 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="716ddd13-6b20-409e-aa1e-f0ea2d4cbdd5" containerName="manager" Jul 13 22:07:08.237890 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:08.237873 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s7x6w" Jul 13 22:07:08.252129 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:07:08.252084 2555 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"51845634-9547-44e3-acdc-030c90cad1cc\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-859d7b78bd-9fbr8\": pods \"limitador-operator-controller-manager-859d7b78bd-9fbr8\" not found" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9fbr8" Jul 13 22:07:08.254391 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:07:08.254366 2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-xc68s\" is forbidden: User \"system:node:ip-10-0-133-234.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-234.ec2.internal' and this object" podUID="716ddd13-6b20-409e-aa1e-f0ea2d4cbdd5" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xc68s" Jul 13 22:07:08.260684 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:08.260658 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s7x6w"] Jul 13 22:07:08.271626 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:08.271600 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6jqnv"] Jul 13 22:07:08.275004 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:08.274987 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6jqnv" Jul 13 22:07:08.298782 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:08.298759 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z27n6\" (UniqueName: \"kubernetes.io/projected/64d3da49-1790-4547-a1ee-1000b8bd3570-kube-api-access-z27n6\") pod \"kuadrant-operator-controller-manager-8684c9ff49-s7x6w\" (UID: \"64d3da49-1790-4547-a1ee-1000b8bd3570\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s7x6w" Jul 13 22:07:08.298895 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:08.298877 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/64d3da49-1790-4547-a1ee-1000b8bd3570-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-s7x6w\" (UID: \"64d3da49-1790-4547-a1ee-1000b8bd3570\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s7x6w" Jul 13 22:07:08.305052 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:08.305029 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6jqnv"] Jul 13 22:07:08.323407 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:07:08.323364 2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-9fbr8\" is forbidden: User \"system:node:ip-10-0-133-234.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-234.ec2.internal' and this object" podUID="51845634-9547-44e3-acdc-030c90cad1cc" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9fbr8" Jul 13 22:07:08.365712 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:07:08.365670 2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-xc68s\" is forbidden: User \"system:node:ip-10-0-133-234.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-234.ec2.internal' and this object" podUID="716ddd13-6b20-409e-aa1e-f0ea2d4cbdd5" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xc68s" Jul 13 22:07:08.369659 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:07:08.369631 2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-9fbr8\" is forbidden: User \"system:node:ip-10-0-133-234.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-234.ec2.internal' and this object" podUID="51845634-9547-44e3-acdc-030c90cad1cc" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9fbr8" Jul 13 22:07:08.399498 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:08.399471 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4xjt\" (UniqueName: \"kubernetes.io/projected/2851b3be-0211-4fc2-85c0-b1dca7ea138c-kube-api-access-f4xjt\") pod \"limitador-operator-controller-manager-859d7b78bd-6jqnv\" (UID: \"2851b3be-0211-4fc2-85c0-b1dca7ea138c\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6jqnv" Jul 13 22:07:08.399677 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:08.399533 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/64d3da49-1790-4547-a1ee-1000b8bd3570-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-s7x6w\" (UID: \"64d3da49-1790-4547-a1ee-1000b8bd3570\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s7x6w" Jul 13 22:07:08.399677 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:08.399610 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z27n6\" (UniqueName: \"kubernetes.io/projected/64d3da49-1790-4547-a1ee-1000b8bd3570-kube-api-access-z27n6\") pod \"kuadrant-operator-controller-manager-8684c9ff49-s7x6w\" (UID: \"64d3da49-1790-4547-a1ee-1000b8bd3570\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s7x6w" Jul 13 22:07:08.399911 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:08.399891 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/64d3da49-1790-4547-a1ee-1000b8bd3570-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-s7x6w\" (UID: \"64d3da49-1790-4547-a1ee-1000b8bd3570\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s7x6w" Jul 13 22:07:08.419995 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:08.419968 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z27n6\" (UniqueName: \"kubernetes.io/projected/64d3da49-1790-4547-a1ee-1000b8bd3570-kube-api-access-z27n6\") pod \"kuadrant-operator-controller-manager-8684c9ff49-s7x6w\" (UID: \"64d3da49-1790-4547-a1ee-1000b8bd3570\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s7x6w" Jul 13 22:07:08.436466 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:08.436403 2555 generic.go:356] "Generic (PLEG): container finished" podID="716ddd13-6b20-409e-aa1e-f0ea2d4cbdd5" containerID="8d472b2417f41e4e23a4414477b9764faeb32cfa9e973166aff6d24c4c4d47e9" exitCode=0 Jul 13 22:07:08.438377 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:08.438341 2555 generic.go:356] "Generic (PLEG): container finished" podID="51845634-9547-44e3-acdc-030c90cad1cc" containerID="896e47dc67871e3a75aaee619c26a197531e50fbc6ce4d05be16e59c2bd190ff" exitCode=0 Jul 13 22:07:08.461035 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:08.461015 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xc68s" Jul 13 22:07:08.463565 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:07:08.463541 2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-xc68s\" is forbidden: User \"system:node:ip-10-0-133-234.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-234.ec2.internal' and this object" podUID="716ddd13-6b20-409e-aa1e-f0ea2d4cbdd5" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xc68s" Jul 13 22:07:08.463798 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:08.463784 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9fbr8" Jul 13 22:07:08.466013 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:07:08.465991 2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-9fbr8\" is forbidden: User \"system:node:ip-10-0-133-234.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-234.ec2.internal' and this object" podUID="51845634-9547-44e3-acdc-030c90cad1cc" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9fbr8" Jul 13 22:07:08.468021 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:07:08.468001 2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-xc68s\" is forbidden: User \"system:node:ip-10-0-133-234.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-234.ec2.internal' and this object" podUID="716ddd13-6b20-409e-aa1e-f0ea2d4cbdd5" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xc68s" Jul 13 22:07:08.469948 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:07:08.469927 2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-9fbr8\" is forbidden: User \"system:node:ip-10-0-133-234.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-234.ec2.internal' and this object" podUID="51845634-9547-44e3-acdc-030c90cad1cc" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9fbr8" Jul 13 22:07:08.500374 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:08.500348 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/716ddd13-6b20-409e-aa1e-f0ea2d4cbdd5-extensions-socket-volume\") pod \"716ddd13-6b20-409e-aa1e-f0ea2d4cbdd5\" (UID: \"716ddd13-6b20-409e-aa1e-f0ea2d4cbdd5\") " Jul 13 22:07:08.500448 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:08.500416 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jcr5s\" (UniqueName: \"kubernetes.io/projected/716ddd13-6b20-409e-aa1e-f0ea2d4cbdd5-kube-api-access-jcr5s\") pod \"716ddd13-6b20-409e-aa1e-f0ea2d4cbdd5\" (UID: \"716ddd13-6b20-409e-aa1e-f0ea2d4cbdd5\") " Jul 13 22:07:08.500448 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:08.500432 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k65d2\" (UniqueName: \"kubernetes.io/projected/51845634-9547-44e3-acdc-030c90cad1cc-kube-api-access-k65d2\") pod \"51845634-9547-44e3-acdc-030c90cad1cc\" (UID: \"51845634-9547-44e3-acdc-030c90cad1cc\") " Jul 13 22:07:08.500528 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:08.500510 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4xjt\" (UniqueName: \"kubernetes.io/projected/2851b3be-0211-4fc2-85c0-b1dca7ea138c-kube-api-access-f4xjt\") pod \"limitador-operator-controller-manager-859d7b78bd-6jqnv\" (UID: \"2851b3be-0211-4fc2-85c0-b1dca7ea138c\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6jqnv" Jul 13 22:07:08.500796 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:08.500756 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/716ddd13-6b20-409e-aa1e-f0ea2d4cbdd5-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "716ddd13-6b20-409e-aa1e-f0ea2d4cbdd5" (UID: "716ddd13-6b20-409e-aa1e-f0ea2d4cbdd5"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:07:08.502523 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:08.502503 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51845634-9547-44e3-acdc-030c90cad1cc-kube-api-access-k65d2" (OuterVolumeSpecName: "kube-api-access-k65d2") pod "51845634-9547-44e3-acdc-030c90cad1cc" (UID: "51845634-9547-44e3-acdc-030c90cad1cc"). InnerVolumeSpecName "kube-api-access-k65d2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:07:08.502523 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:08.502509 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/716ddd13-6b20-409e-aa1e-f0ea2d4cbdd5-kube-api-access-jcr5s" (OuterVolumeSpecName: "kube-api-access-jcr5s") pod "716ddd13-6b20-409e-aa1e-f0ea2d4cbdd5" (UID: "716ddd13-6b20-409e-aa1e-f0ea2d4cbdd5"). InnerVolumeSpecName "kube-api-access-jcr5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:07:08.514644 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:08.514623 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4xjt\" (UniqueName: \"kubernetes.io/projected/2851b3be-0211-4fc2-85c0-b1dca7ea138c-kube-api-access-f4xjt\") pod \"limitador-operator-controller-manager-859d7b78bd-6jqnv\" (UID: \"2851b3be-0211-4fc2-85c0-b1dca7ea138c\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6jqnv" Jul 13 22:07:08.559672 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:08.559645 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51845634-9547-44e3-acdc-030c90cad1cc" path="/var/lib/kubelet/pods/51845634-9547-44e3-acdc-030c90cad1cc/volumes" Jul 13 22:07:08.559977 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:08.559964 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="716ddd13-6b20-409e-aa1e-f0ea2d4cbdd5" path="/var/lib/kubelet/pods/716ddd13-6b20-409e-aa1e-f0ea2d4cbdd5/volumes" Jul 13 22:07:08.599805 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:08.599781 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s7x6w" Jul 13 22:07:08.601428 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:08.601412 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcr5s\" (UniqueName: \"kubernetes.io/projected/716ddd13-6b20-409e-aa1e-f0ea2d4cbdd5-kube-api-access-jcr5s\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:07:08.601489 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:08.601431 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k65d2\" (UniqueName: \"kubernetes.io/projected/51845634-9547-44e3-acdc-030c90cad1cc-kube-api-access-k65d2\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:07:08.601489 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:08.601440 2555 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/716ddd13-6b20-409e-aa1e-f0ea2d4cbdd5-extensions-socket-volume\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:07:08.606623 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:08.606603 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6jqnv" Jul 13 22:07:08.747815 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:08.747784 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s7x6w"] Jul 13 22:07:08.751400 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:07:08.751372 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64d3da49_1790_4547_a1ee_1000b8bd3570.slice/crio-caa6a3d2a0ac12a5f7103ea9fe6320f94d706b7704409d86eda026cc93945da4 WatchSource:0}: Error finding container caa6a3d2a0ac12a5f7103ea9fe6320f94d706b7704409d86eda026cc93945da4: Status 404 returned error can't find the container with id caa6a3d2a0ac12a5f7103ea9fe6320f94d706b7704409d86eda026cc93945da4 Jul 13 22:07:08.765539 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:08.765517 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6jqnv"] Jul 13 22:07:08.766775 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:07:08.766750 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2851b3be_0211_4fc2_85c0_b1dca7ea138c.slice/crio-0053115f65c547f226e94e6adbefc9090fa52cc54a8deba978b1e69c6df44794 WatchSource:0}: Error finding container 0053115f65c547f226e94e6adbefc9090fa52cc54a8deba978b1e69c6df44794: Status 404 returned error can't find the container with id 0053115f65c547f226e94e6adbefc9090fa52cc54a8deba978b1e69c6df44794 Jul 13 22:07:09.444964 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:09.444927 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6jqnv" event={"ID":"2851b3be-0211-4fc2-85c0-b1dca7ea138c","Type":"ContainerStarted","Data":"046c8161f221e4fa0c09a5e9efaa20faea6576fa1cbab4c8c3e1b4f97cfcf7a9"} Jul 13 22:07:09.445408 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:09.444991 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6jqnv" event={"ID":"2851b3be-0211-4fc2-85c0-b1dca7ea138c","Type":"ContainerStarted","Data":"0053115f65c547f226e94e6adbefc9090fa52cc54a8deba978b1e69c6df44794"} Jul 13 22:07:09.445408 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:09.445019 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6jqnv" Jul 13 22:07:09.446334 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:09.446303 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xc68s" Jul 13 22:07:09.446464 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:09.446345 2555 scope.go:117] "RemoveContainer" containerID="8d472b2417f41e4e23a4414477b9764faeb32cfa9e973166aff6d24c4c4d47e9" Jul 13 22:07:09.448144 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:09.448116 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s7x6w" event={"ID":"64d3da49-1790-4547-a1ee-1000b8bd3570","Type":"ContainerStarted","Data":"be4752ce8836b0ea47a5d890457edf3c83848a94f3b3f2476f184b729605063a"} Jul 13 22:07:09.448257 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:09.448172 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s7x6w" event={"ID":"64d3da49-1790-4547-a1ee-1000b8bd3570","Type":"ContainerStarted","Data":"caa6a3d2a0ac12a5f7103ea9fe6320f94d706b7704409d86eda026cc93945da4"} Jul 13 22:07:09.449755 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:09.449706 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9fbr8" Jul 13 22:07:09.455621 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:09.455603 2555 scope.go:117] "RemoveContainer" containerID="896e47dc67871e3a75aaee619c26a197531e50fbc6ce4d05be16e59c2bd190ff" Jul 13 22:07:09.469951 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:09.469910 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6jqnv" podStartSLOduration=1.469897412 podStartE2EDuration="1.469897412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:07:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:07:09.46828107 +0000 UTC m=+435.462035384" watchObservedRunningTime="2026-07-13 22:07:09.469897412 +0000 UTC m=+435.463651706" Jul 13 22:07:09.496409 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:09.496356 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s7x6w" podStartSLOduration=1.496340692 podStartE2EDuration="1.496340692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:07:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:07:09.493802208 +0000 UTC m=+435.487556527" watchObservedRunningTime="2026-07-13 22:07:09.496340692 +0000 UTC m=+435.490094988" Jul 13 22:07:09.583709 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:09.583674 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5wxmf"] Jul 13 22:07:09.588533 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:09.588513 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5wxmf" Jul 13 22:07:09.606965 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:09.606932 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5wxmf"] Jul 13 22:07:09.712829 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:09.712733 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtgmm\" (UniqueName: \"kubernetes.io/projected/23dbc2e4-e4d6-446c-93fe-4212b860f025-kube-api-access-wtgmm\") pod \"kuadrant-operator-controller-manager-86878c769f-5wxmf\" (UID: \"23dbc2e4-e4d6-446c-93fe-4212b860f025\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5wxmf" Jul 13 22:07:09.712829 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:09.712799 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/23dbc2e4-e4d6-446c-93fe-4212b860f025-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-5wxmf\" (UID: \"23dbc2e4-e4d6-446c-93fe-4212b860f025\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5wxmf" Jul 13 22:07:09.814129 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:09.814091 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/23dbc2e4-e4d6-446c-93fe-4212b860f025-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-5wxmf\" (UID: \"23dbc2e4-e4d6-446c-93fe-4212b860f025\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5wxmf" Jul 13 22:07:09.814304 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:09.814170 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtgmm\" (UniqueName: \"kubernetes.io/projected/23dbc2e4-e4d6-446c-93fe-4212b860f025-kube-api-access-wtgmm\") pod \"kuadrant-operator-controller-manager-86878c769f-5wxmf\" (UID: \"23dbc2e4-e4d6-446c-93fe-4212b860f025\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5wxmf" Jul 13 22:07:09.814487 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:09.814468 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/23dbc2e4-e4d6-446c-93fe-4212b860f025-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-5wxmf\" (UID: \"23dbc2e4-e4d6-446c-93fe-4212b860f025\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5wxmf" Jul 13 22:07:09.826659 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:09.826634 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtgmm\" (UniqueName: \"kubernetes.io/projected/23dbc2e4-e4d6-446c-93fe-4212b860f025-kube-api-access-wtgmm\") pod \"kuadrant-operator-controller-manager-86878c769f-5wxmf\" (UID: \"23dbc2e4-e4d6-446c-93fe-4212b860f025\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5wxmf" Jul 13 22:07:09.899526 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:09.899496 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5wxmf" Jul 13 22:07:10.044329 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:10.044300 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5wxmf"] Jul 13 22:07:10.047770 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:07:10.047730 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23dbc2e4_e4d6_446c_93fe_4212b860f025.slice/crio-14c0bfe12e63a3d59a103fbafc60fd5487e9fca1c08a7c463b58eb31e5afc0f4 WatchSource:0}: Error finding container 14c0bfe12e63a3d59a103fbafc60fd5487e9fca1c08a7c463b58eb31e5afc0f4: Status 404 returned error can't find the container with id 14c0bfe12e63a3d59a103fbafc60fd5487e9fca1c08a7c463b58eb31e5afc0f4 Jul 13 22:07:10.457511 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:10.457472 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5wxmf" event={"ID":"23dbc2e4-e4d6-446c-93fe-4212b860f025","Type":"ContainerStarted","Data":"ee10a5fb4e6dc2d5c38b88a7cd38ad8ce7df7176dae056bd8cef78fe02198bf7"} Jul 13 22:07:10.457887 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:10.457534 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5wxmf" event={"ID":"23dbc2e4-e4d6-446c-93fe-4212b860f025","Type":"ContainerStarted","Data":"14c0bfe12e63a3d59a103fbafc60fd5487e9fca1c08a7c463b58eb31e5afc0f4"} Jul 13 22:07:10.457887 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:10.457627 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s7x6w" Jul 13 22:07:10.480786 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:10.480740 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5wxmf" podStartSLOduration=1.4807244019999999 podStartE2EDuration="1.480724402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:07:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:07:10.480269019 +0000 UTC m=+436.474023321" watchObservedRunningTime="2026-07-13 22:07:10.480724402 +0000 UTC m=+436.474478697" Jul 13 22:07:11.462821 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:11.462786 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5wxmf" Jul 13 22:07:20.459774 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:20.459701 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6jqnv" Jul 13 22:07:21.465694 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:21.465665 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s7x6w" Jul 13 22:07:22.470360 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:22.470329 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5wxmf" Jul 13 22:07:22.537723 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:22.537692 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s7x6w"] Jul 13 22:07:22.537928 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:22.537905 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s7x6w" podUID="64d3da49-1790-4547-a1ee-1000b8bd3570" containerName="manager" containerID="cri-o://be4752ce8836b0ea47a5d890457edf3c83848a94f3b3f2476f184b729605063a" gracePeriod=10 Jul 13 22:07:22.788502 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:22.788479 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s7x6w" Jul 13 22:07:22.923922 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:22.923899 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/64d3da49-1790-4547-a1ee-1000b8bd3570-extensions-socket-volume\") pod \"64d3da49-1790-4547-a1ee-1000b8bd3570\" (UID: \"64d3da49-1790-4547-a1ee-1000b8bd3570\") " Jul 13 22:07:22.924063 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:22.923968 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z27n6\" (UniqueName: \"kubernetes.io/projected/64d3da49-1790-4547-a1ee-1000b8bd3570-kube-api-access-z27n6\") pod \"64d3da49-1790-4547-a1ee-1000b8bd3570\" (UID: \"64d3da49-1790-4547-a1ee-1000b8bd3570\") " Jul 13 22:07:22.924343 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:22.924311 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64d3da49-1790-4547-a1ee-1000b8bd3570-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "64d3da49-1790-4547-a1ee-1000b8bd3570" (UID: "64d3da49-1790-4547-a1ee-1000b8bd3570"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:07:22.926059 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:22.926031 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64d3da49-1790-4547-a1ee-1000b8bd3570-kube-api-access-z27n6" (OuterVolumeSpecName: "kube-api-access-z27n6") pod "64d3da49-1790-4547-a1ee-1000b8bd3570" (UID: "64d3da49-1790-4547-a1ee-1000b8bd3570"). InnerVolumeSpecName "kube-api-access-z27n6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:07:23.025293 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:23.025157 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z27n6\" (UniqueName: \"kubernetes.io/projected/64d3da49-1790-4547-a1ee-1000b8bd3570-kube-api-access-z27n6\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:07:23.025293 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:23.025194 2555 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/64d3da49-1790-4547-a1ee-1000b8bd3570-extensions-socket-volume\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:07:23.526671 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:23.526634 2555 generic.go:356] "Generic (PLEG): container finished" podID="64d3da49-1790-4547-a1ee-1000b8bd3570" containerID="be4752ce8836b0ea47a5d890457edf3c83848a94f3b3f2476f184b729605063a" exitCode=0 Jul 13 22:07:23.526671 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:23.526673 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s7x6w" Jul 13 22:07:23.527123 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:23.526722 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s7x6w" event={"ID":"64d3da49-1790-4547-a1ee-1000b8bd3570","Type":"ContainerDied","Data":"be4752ce8836b0ea47a5d890457edf3c83848a94f3b3f2476f184b729605063a"} Jul 13 22:07:23.527123 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:23.526776 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s7x6w" event={"ID":"64d3da49-1790-4547-a1ee-1000b8bd3570","Type":"ContainerDied","Data":"caa6a3d2a0ac12a5f7103ea9fe6320f94d706b7704409d86eda026cc93945da4"} Jul 13 22:07:23.527123 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:23.526802 2555 scope.go:117] "RemoveContainer" containerID="be4752ce8836b0ea47a5d890457edf3c83848a94f3b3f2476f184b729605063a" Jul 13 22:07:23.537789 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:23.537372 2555 scope.go:117] "RemoveContainer" containerID="be4752ce8836b0ea47a5d890457edf3c83848a94f3b3f2476f184b729605063a" Jul 13 22:07:23.537885 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:07:23.537789 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be4752ce8836b0ea47a5d890457edf3c83848a94f3b3f2476f184b729605063a\": container with ID starting with be4752ce8836b0ea47a5d890457edf3c83848a94f3b3f2476f184b729605063a not found: ID does not exist" containerID="be4752ce8836b0ea47a5d890457edf3c83848a94f3b3f2476f184b729605063a" Jul 13 22:07:23.537885 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:23.537824 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be4752ce8836b0ea47a5d890457edf3c83848a94f3b3f2476f184b729605063a"} err="failed to get container status \"be4752ce8836b0ea47a5d890457edf3c83848a94f3b3f2476f184b729605063a\": rpc error: code = NotFound desc = could not find container \"be4752ce8836b0ea47a5d890457edf3c83848a94f3b3f2476f184b729605063a\": container with ID starting with be4752ce8836b0ea47a5d890457edf3c83848a94f3b3f2476f184b729605063a not found: ID does not exist" Jul 13 22:07:23.558696 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:23.558642 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s7x6w"] Jul 13 22:07:23.560438 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:23.560419 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s7x6w"] Jul 13 22:07:24.558568 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:24.558534 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64d3da49-1790-4547-a1ee-1000b8bd3570" path="/var/lib/kubelet/pods/64d3da49-1790-4547-a1ee-1000b8bd3570/volumes" Jul 13 22:07:38.902011 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:38.901980 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jhf2p"] Jul 13 22:07:38.902434 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:38.902341 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64d3da49-1790-4547-a1ee-1000b8bd3570" containerName="manager" Jul 13 22:07:38.902434 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:38.902352 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="64d3da49-1790-4547-a1ee-1000b8bd3570" containerName="manager" Jul 13 22:07:38.902434 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:38.902401 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="64d3da49-1790-4547-a1ee-1000b8bd3570" containerName="manager" Jul 13 22:07:38.906461 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:38.906444 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jhf2p" Jul 13 22:07:38.909262 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:38.909239 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-2wmm5\"" Jul 13 22:07:38.921666 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:38.921645 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jhf2p"] Jul 13 22:07:39.058444 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:39.058407 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/504e7d67-a82c-4653-bb50-a64a7a240eb5-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-jhf2p\" (UID: \"504e7d67-a82c-4653-bb50-a64a7a240eb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jhf2p" Jul 13 22:07:39.058646 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:39.058455 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/504e7d67-a82c-4653-bb50-a64a7a240eb5-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-jhf2p\" (UID: \"504e7d67-a82c-4653-bb50-a64a7a240eb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jhf2p" Jul 13 22:07:39.058646 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:39.058476 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/504e7d67-a82c-4653-bb50-a64a7a240eb5-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-jhf2p\" (UID: \"504e7d67-a82c-4653-bb50-a64a7a240eb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jhf2p" Jul 13 22:07:39.058646 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:39.058491 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/504e7d67-a82c-4653-bb50-a64a7a240eb5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-jhf2p\" (UID: \"504e7d67-a82c-4653-bb50-a64a7a240eb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jhf2p" Jul 13 22:07:39.058646 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:39.058566 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/504e7d67-a82c-4653-bb50-a64a7a240eb5-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-jhf2p\" (UID: \"504e7d67-a82c-4653-bb50-a64a7a240eb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jhf2p" Jul 13 22:07:39.058646 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:39.058612 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/504e7d67-a82c-4653-bb50-a64a7a240eb5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-jhf2p\" (UID: \"504e7d67-a82c-4653-bb50-a64a7a240eb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jhf2p" Jul 13 22:07:39.058646 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:39.058646 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/504e7d67-a82c-4653-bb50-a64a7a240eb5-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-jhf2p\" (UID: \"504e7d67-a82c-4653-bb50-a64a7a240eb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jhf2p" Jul 13 22:07:39.058916 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:39.058728 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/504e7d67-a82c-4653-bb50-a64a7a240eb5-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-jhf2p\" (UID: \"504e7d67-a82c-4653-bb50-a64a7a240eb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jhf2p" Jul 13 22:07:39.058916 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:39.058776 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf274\" (UniqueName: \"kubernetes.io/projected/504e7d67-a82c-4653-bb50-a64a7a240eb5-kube-api-access-vf274\") pod \"maas-default-gateway-openshift-default-56654dcf7-jhf2p\" (UID: \"504e7d67-a82c-4653-bb50-a64a7a240eb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jhf2p" Jul 13 22:07:39.159662 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:39.159572 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf274\" (UniqueName: \"kubernetes.io/projected/504e7d67-a82c-4653-bb50-a64a7a240eb5-kube-api-access-vf274\") pod \"maas-default-gateway-openshift-default-56654dcf7-jhf2p\" (UID: \"504e7d67-a82c-4653-bb50-a64a7a240eb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jhf2p" Jul 13 22:07:39.159662 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:39.159625 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/504e7d67-a82c-4653-bb50-a64a7a240eb5-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-jhf2p\" (UID: \"504e7d67-a82c-4653-bb50-a64a7a240eb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jhf2p" Jul 13 22:07:39.159888 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:39.159678 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/504e7d67-a82c-4653-bb50-a64a7a240eb5-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-jhf2p\" (UID: \"504e7d67-a82c-4653-bb50-a64a7a240eb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jhf2p" Jul 13 22:07:39.159888 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:39.159706 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/504e7d67-a82c-4653-bb50-a64a7a240eb5-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-jhf2p\" (UID: \"504e7d67-a82c-4653-bb50-a64a7a240eb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jhf2p" Jul 13 22:07:39.159888 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:39.159723 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/504e7d67-a82c-4653-bb50-a64a7a240eb5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-jhf2p\" (UID: \"504e7d67-a82c-4653-bb50-a64a7a240eb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jhf2p" Jul 13 22:07:39.159888 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:39.159775 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/504e7d67-a82c-4653-bb50-a64a7a240eb5-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-jhf2p\" (UID: \"504e7d67-a82c-4653-bb50-a64a7a240eb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jhf2p" Jul 13 22:07:39.159888 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:39.159810 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/504e7d67-a82c-4653-bb50-a64a7a240eb5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-jhf2p\" (UID: \"504e7d67-a82c-4653-bb50-a64a7a240eb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jhf2p" Jul 13 22:07:39.159888 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:39.159831 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/504e7d67-a82c-4653-bb50-a64a7a240eb5-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-jhf2p\" (UID: \"504e7d67-a82c-4653-bb50-a64a7a240eb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jhf2p" Jul 13 22:07:39.160245 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:39.159891 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/504e7d67-a82c-4653-bb50-a64a7a240eb5-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-jhf2p\" (UID: \"504e7d67-a82c-4653-bb50-a64a7a240eb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jhf2p" Jul 13 22:07:39.160245 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:39.160108 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/504e7d67-a82c-4653-bb50-a64a7a240eb5-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-jhf2p\" (UID: \"504e7d67-a82c-4653-bb50-a64a7a240eb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jhf2p" Jul 13 22:07:39.160245 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:39.160139 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/504e7d67-a82c-4653-bb50-a64a7a240eb5-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-jhf2p\" (UID: \"504e7d67-a82c-4653-bb50-a64a7a240eb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jhf2p" Jul 13 22:07:39.160450 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:39.160297 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/504e7d67-a82c-4653-bb50-a64a7a240eb5-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-jhf2p\" (UID: \"504e7d67-a82c-4653-bb50-a64a7a240eb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jhf2p" Jul 13 22:07:39.160450 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:39.160308 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/504e7d67-a82c-4653-bb50-a64a7a240eb5-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-jhf2p\" (UID: \"504e7d67-a82c-4653-bb50-a64a7a240eb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jhf2p" Jul 13 22:07:39.160643 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:39.160620 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/504e7d67-a82c-4653-bb50-a64a7a240eb5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-jhf2p\" (UID: \"504e7d67-a82c-4653-bb50-a64a7a240eb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jhf2p" Jul 13 22:07:39.162241 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:39.162218 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/504e7d67-a82c-4653-bb50-a64a7a240eb5-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-jhf2p\" (UID: \"504e7d67-a82c-4653-bb50-a64a7a240eb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jhf2p" Jul 13 22:07:39.162355 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:39.162337 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/504e7d67-a82c-4653-bb50-a64a7a240eb5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-jhf2p\" (UID: \"504e7d67-a82c-4653-bb50-a64a7a240eb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jhf2p" Jul 13 22:07:39.168213 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:39.168174 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/504e7d67-a82c-4653-bb50-a64a7a240eb5-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-jhf2p\" (UID: \"504e7d67-a82c-4653-bb50-a64a7a240eb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jhf2p" Jul 13 22:07:39.171140 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:39.171117 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf274\" (UniqueName: \"kubernetes.io/projected/504e7d67-a82c-4653-bb50-a64a7a240eb5-kube-api-access-vf274\") pod \"maas-default-gateway-openshift-default-56654dcf7-jhf2p\" (UID: \"504e7d67-a82c-4653-bb50-a64a7a240eb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jhf2p" Jul 13 22:07:39.218741 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:39.218712 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jhf2p" Jul 13 22:07:39.358017 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:39.357980 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jhf2p"] Jul 13 22:07:39.359594 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:07:39.359566 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod504e7d67_a82c_4653_bb50_a64a7a240eb5.slice/crio-0592e5a36329c5d13ab193f4b634d89d85ff7020c120823ad62fd7119774646e WatchSource:0}: Error finding container 0592e5a36329c5d13ab193f4b634d89d85ff7020c120823ad62fd7119774646e: Status 404 returned error can't find the container with id 0592e5a36329c5d13ab193f4b634d89d85ff7020c120823ad62fd7119774646e Jul 13 22:07:39.361633 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:39.361603 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:07:39.361685 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:39.361665 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:07:39.361728 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:39.361693 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:07:39.607489 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:39.607450 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jhf2p" event={"ID":"504e7d67-a82c-4653-bb50-a64a7a240eb5","Type":"ContainerStarted","Data":"c65cea25b749181159f7ae7a1b8ec11067abaa6fb8d9fe79853b86e758f56209"} Jul 13 22:07:39.607669 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:39.607516 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jhf2p" event={"ID":"504e7d67-a82c-4653-bb50-a64a7a240eb5","Type":"ContainerStarted","Data":"0592e5a36329c5d13ab193f4b634d89d85ff7020c120823ad62fd7119774646e"} Jul 13 22:07:39.631804 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:39.631761 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jhf2p" podStartSLOduration=1.631747963 podStartE2EDuration="1.631747963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:07:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:07:39.631283313 +0000 UTC m=+465.625037634" watchObservedRunningTime="2026-07-13 22:07:39.631747963 +0000 UTC m=+465.625502256" Jul 13 22:07:40.219282 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:40.219243 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jhf2p" Jul 13 22:07:40.223884 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:40.223861 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jhf2p" Jul 13 22:07:40.612835 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:40.612746 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jhf2p" Jul 13 22:07:40.613845 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:40.613818 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jhf2p" Jul 13 22:07:41.419472 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:41.419432 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-prnzg"] Jul 13 22:07:41.422256 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:41.422237 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-prnzg" Jul 13 22:07:41.425184 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:41.425165 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-s2kgp\"" Jul 13 22:07:41.433814 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:41.433776 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-prnzg"] Jul 13 22:07:41.585872 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:41.585833 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76gpd\" (UniqueName: \"kubernetes.io/projected/23f8ea30-70e0-46b5-93f3-466564fa4999-kube-api-access-76gpd\") pod \"authorino-58cd955675-prnzg\" (UID: \"23f8ea30-70e0-46b5-93f3-466564fa4999\") " pod="kuadrant-system/authorino-58cd955675-prnzg" Jul 13 22:07:41.687378 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:41.687272 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76gpd\" (UniqueName: \"kubernetes.io/projected/23f8ea30-70e0-46b5-93f3-466564fa4999-kube-api-access-76gpd\") pod \"authorino-58cd955675-prnzg\" (UID: \"23f8ea30-70e0-46b5-93f3-466564fa4999\") " pod="kuadrant-system/authorino-58cd955675-prnzg" Jul 13 22:07:41.696702 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:41.696669 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76gpd\" (UniqueName: \"kubernetes.io/projected/23f8ea30-70e0-46b5-93f3-466564fa4999-kube-api-access-76gpd\") pod \"authorino-58cd955675-prnzg\" (UID: \"23f8ea30-70e0-46b5-93f3-466564fa4999\") " pod="kuadrant-system/authorino-58cd955675-prnzg" Jul 13 22:07:41.731938 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:41.731908 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-prnzg" Jul 13 22:07:41.885361 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:07:41.885319 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod23f8ea30_70e0_46b5_93f3_466564fa4999.slice/crio-73df3a76c6e570cf57658992daf40de831009ef1ea2c81efb33de28857d09e87 WatchSource:0}: Error finding container 73df3a76c6e570cf57658992daf40de831009ef1ea2c81efb33de28857d09e87: Status 404 returned error can't find the container with id 73df3a76c6e570cf57658992daf40de831009ef1ea2c81efb33de28857d09e87 Jul 13 22:07:41.885582 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:41.885563 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-prnzg"] Jul 13 22:07:42.627096 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:42.627002 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-prnzg" event={"ID":"23f8ea30-70e0-46b5-93f3-466564fa4999","Type":"ContainerStarted","Data":"73df3a76c6e570cf57658992daf40de831009ef1ea2c81efb33de28857d09e87"} Jul 13 22:07:45.654621 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:45.654581 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-prnzg" event={"ID":"23f8ea30-70e0-46b5-93f3-466564fa4999","Type":"ContainerStarted","Data":"010e7c984798ea784ce17c7f5e948d6f968d51a7ee9aca7f15132e5f0606b73a"} Jul 13 22:07:45.675499 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:45.675454 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-prnzg" podStartSLOduration=1.370165579 podStartE2EDuration="4.67544032s" totalImagesPullingTime="3.305274741s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:07:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:07:45.673785555 +0000 UTC m=+471.667539848" watchObservedRunningTime="2026-07-13 22:07:45.67544032 +0000 UTC m=+471.669194612" Jul 13 22:07:48.355796 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:48.355759 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-prnzg"] Jul 13 22:07:48.356218 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:48.355909 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-prnzg" podUID="23f8ea30-70e0-46b5-93f3-466564fa4999" containerName="authorino" containerID="cri-o://010e7c984798ea784ce17c7f5e948d6f968d51a7ee9aca7f15132e5f0606b73a" gracePeriod=30 Jul 13 22:07:48.607363 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:48.607293 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-prnzg" Jul 13 22:07:48.648321 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:48.648298 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-76gpd\" (UniqueName: \"kubernetes.io/projected/23f8ea30-70e0-46b5-93f3-466564fa4999-kube-api-access-76gpd\") pod \"23f8ea30-70e0-46b5-93f3-466564fa4999\" (UID: \"23f8ea30-70e0-46b5-93f3-466564fa4999\") " Jul 13 22:07:48.650490 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:48.650462 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23f8ea30-70e0-46b5-93f3-466564fa4999-kube-api-access-76gpd" (OuterVolumeSpecName: "kube-api-access-76gpd") pod "23f8ea30-70e0-46b5-93f3-466564fa4999" (UID: "23f8ea30-70e0-46b5-93f3-466564fa4999"). InnerVolumeSpecName "kube-api-access-76gpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:07:48.670773 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:48.670750 2555 generic.go:356] "Generic (PLEG): container finished" podID="23f8ea30-70e0-46b5-93f3-466564fa4999" containerID="010e7c984798ea784ce17c7f5e948d6f968d51a7ee9aca7f15132e5f0606b73a" exitCode=0 Jul 13 22:07:48.670874 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:48.670793 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-prnzg" Jul 13 22:07:48.670874 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:48.670802 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-prnzg" event={"ID":"23f8ea30-70e0-46b5-93f3-466564fa4999","Type":"ContainerDied","Data":"010e7c984798ea784ce17c7f5e948d6f968d51a7ee9aca7f15132e5f0606b73a"} Jul 13 22:07:48.670874 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:48.670832 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-prnzg" event={"ID":"23f8ea30-70e0-46b5-93f3-466564fa4999","Type":"ContainerDied","Data":"73df3a76c6e570cf57658992daf40de831009ef1ea2c81efb33de28857d09e87"} Jul 13 22:07:48.670874 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:48.670848 2555 scope.go:117] "RemoveContainer" containerID="010e7c984798ea784ce17c7f5e948d6f968d51a7ee9aca7f15132e5f0606b73a" Jul 13 22:07:48.680177 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:48.680159 2555 scope.go:117] "RemoveContainer" containerID="010e7c984798ea784ce17c7f5e948d6f968d51a7ee9aca7f15132e5f0606b73a" Jul 13 22:07:48.680449 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:07:48.680433 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"010e7c984798ea784ce17c7f5e948d6f968d51a7ee9aca7f15132e5f0606b73a\": container with ID starting with 010e7c984798ea784ce17c7f5e948d6f968d51a7ee9aca7f15132e5f0606b73a not found: ID does not exist" containerID="010e7c984798ea784ce17c7f5e948d6f968d51a7ee9aca7f15132e5f0606b73a" Jul 13 22:07:48.680503 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:48.680458 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"010e7c984798ea784ce17c7f5e948d6f968d51a7ee9aca7f15132e5f0606b73a"} err="failed to get container status \"010e7c984798ea784ce17c7f5e948d6f968d51a7ee9aca7f15132e5f0606b73a\": rpc error: code = NotFound desc = could not find container \"010e7c984798ea784ce17c7f5e948d6f968d51a7ee9aca7f15132e5f0606b73a\": container with ID starting with 010e7c984798ea784ce17c7f5e948d6f968d51a7ee9aca7f15132e5f0606b73a not found: ID does not exist" Jul 13 22:07:48.692271 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:48.692251 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-prnzg"] Jul 13 22:07:48.704054 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:48.704032 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-prnzg"] Jul 13 22:07:48.749593 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:48.749558 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76gpd\" (UniqueName: \"kubernetes.io/projected/23f8ea30-70e0-46b5-93f3-466564fa4999-kube-api-access-76gpd\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:07:50.559614 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:07:50.559582 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23f8ea30-70e0-46b5-93f3-466564fa4999" path="/var/lib/kubelet/pods/23f8ea30-70e0-46b5-93f3-466564fa4999/volumes" Jul 13 22:08:15.985952 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:08:15.985914 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-kwcfn"] Jul 13 22:08:15.986470 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:08:15.986312 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23f8ea30-70e0-46b5-93f3-466564fa4999" containerName="authorino" Jul 13 22:08:15.986470 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:08:15.986325 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="23f8ea30-70e0-46b5-93f3-466564fa4999" containerName="authorino" Jul 13 22:08:15.986470 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:08:15.986374 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="23f8ea30-70e0-46b5-93f3-466564fa4999" containerName="authorino" Jul 13 22:08:15.988553 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:08:15.988534 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-kwcfn" Jul 13 22:08:15.991063 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:08:15.991038 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-s2kgp\"" Jul 13 22:08:15.997420 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:08:15.997395 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-kwcfn"] Jul 13 22:08:16.095376 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:08:16.095342 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr68j\" (UniqueName: \"kubernetes.io/projected/a77e7c2c-afde-48e0-9f74-8c2af82c8e85-kube-api-access-lr68j\") pod \"authorino-5d4b67c446-kwcfn\" (UID: \"a77e7c2c-afde-48e0-9f74-8c2af82c8e85\") " pod="kuadrant-system/authorino-5d4b67c446-kwcfn" Jul 13 22:08:16.196822 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:08:16.196794 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr68j\" (UniqueName: \"kubernetes.io/projected/a77e7c2c-afde-48e0-9f74-8c2af82c8e85-kube-api-access-lr68j\") pod \"authorino-5d4b67c446-kwcfn\" (UID: \"a77e7c2c-afde-48e0-9f74-8c2af82c8e85\") " pod="kuadrant-system/authorino-5d4b67c446-kwcfn" Jul 13 22:08:16.208262 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:08:16.208238 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr68j\" (UniqueName: \"kubernetes.io/projected/a77e7c2c-afde-48e0-9f74-8c2af82c8e85-kube-api-access-lr68j\") pod \"authorino-5d4b67c446-kwcfn\" (UID: \"a77e7c2c-afde-48e0-9f74-8c2af82c8e85\") " pod="kuadrant-system/authorino-5d4b67c446-kwcfn" Jul 13 22:08:16.235917 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:08:16.235885 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-kwcfn"] Jul 13 22:08:16.236138 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:08:16.236103 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-kwcfn" Jul 13 22:08:16.264821 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:08:16.264793 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-64bf5445dc-kh6bg"] Jul 13 22:08:16.267158 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:08:16.267140 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-64bf5445dc-kh6bg" Jul 13 22:08:16.280592 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:08:16.280566 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-64bf5445dc-kh6bg"] Jul 13 22:08:16.311764 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:08:16.311737 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-64bf5445dc-kh6bg"] Jul 13 22:08:16.311998 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:08:16.311979 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-6q272], unattached volumes=[], failed to process volumes=[kube-api-access-6q272]: context canceled" pod="kuadrant-system/authorino-64bf5445dc-kh6bg" podUID="ceb42c6b-015a-4c8c-a449-c630c5014c21" Jul 13 22:08:16.343234 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:08:16.342823 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-755b8f8b87-6cqsw"] Jul 13 22:08:16.345539 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:08:16.345514 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-755b8f8b87-6cqsw" Jul 13 22:08:16.348530 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:08:16.348509 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 13 22:08:16.356566 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:08:16.356543 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-755b8f8b87-6cqsw"] Jul 13 22:08:16.376090 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:08:16.376066 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-kwcfn"] Jul 13 22:08:16.378059 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:08:16.378038 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda77e7c2c_afde_48e0_9f74_8c2af82c8e85.slice/crio-d4a7b99820f33fe86cb6d31f802330b9aa082c8f9e6a9fbdc7d7ca94eadd3b1c WatchSource:0}: Error finding container d4a7b99820f33fe86cb6d31f802330b9aa082c8f9e6a9fbdc7d7ca94eadd3b1c: Status 404 returned error can't find the container with id d4a7b99820f33fe86cb6d31f802330b9aa082c8f9e6a9fbdc7d7ca94eadd3b1c Jul 13 22:08:16.398829 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:08:16.398800 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q272\" (UniqueName: \"kubernetes.io/projected/ceb42c6b-015a-4c8c-a449-c630c5014c21-kube-api-access-6q272\") pod \"authorino-64bf5445dc-kh6bg\" (UID: \"ceb42c6b-015a-4c8c-a449-c630c5014c21\") " pod="kuadrant-system/authorino-64bf5445dc-kh6bg" Jul 13 22:08:16.500105 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:08:16.500022 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/65aa5bd7-3f91-4491-8a85-c686955e8e85-tls-cert\") pod \"authorino-755b8f8b87-6cqsw\" (UID: \"65aa5bd7-3f91-4491-8a85-c686955e8e85\") " pod="kuadrant-system/authorino-755b8f8b87-6cqsw" Jul 13 22:08:16.500105 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:08:16.500066 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q272\" (UniqueName: \"kubernetes.io/projected/ceb42c6b-015a-4c8c-a449-c630c5014c21-kube-api-access-6q272\") pod \"authorino-64bf5445dc-kh6bg\" (UID: \"ceb42c6b-015a-4c8c-a449-c630c5014c21\") " pod="kuadrant-system/authorino-64bf5445dc-kh6bg" Jul 13 22:08:16.500319 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:08:16.500196 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2wsx\" (UniqueName: \"kubernetes.io/projected/65aa5bd7-3f91-4491-8a85-c686955e8e85-kube-api-access-q2wsx\") pod \"authorino-755b8f8b87-6cqsw\" (UID: \"65aa5bd7-3f91-4491-8a85-c686955e8e85\") " pod="kuadrant-system/authorino-755b8f8b87-6cqsw" Jul 13 22:08:16.510627 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:08:16.510596 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q272\" (UniqueName: \"kubernetes.io/projected/ceb42c6b-015a-4c8c-a449-c630c5014c21-kube-api-access-6q272\") pod \"authorino-64bf5445dc-kh6bg\" (UID: \"ceb42c6b-015a-4c8c-a449-c630c5014c21\") " pod="kuadrant-system/authorino-64bf5445dc-kh6bg" Jul 13 22:08:16.601382 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:08:16.601347 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/65aa5bd7-3f91-4491-8a85-c686955e8e85-tls-cert\") pod \"authorino-755b8f8b87-6cqsw\" (UID: \"65aa5bd7-3f91-4491-8a85-c686955e8e85\") " pod="kuadrant-system/authorino-755b8f8b87-6cqsw" Jul 13 22:08:16.601547 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:08:16.601420 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2wsx\" (UniqueName: \"kubernetes.io/projected/65aa5bd7-3f91-4491-8a85-c686955e8e85-kube-api-access-q2wsx\") pod \"authorino-755b8f8b87-6cqsw\" (UID: \"65aa5bd7-3f91-4491-8a85-c686955e8e85\") " pod="kuadrant-system/authorino-755b8f8b87-6cqsw" Jul 13 22:08:16.604013 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:08:16.603985 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/65aa5bd7-3f91-4491-8a85-c686955e8e85-tls-cert\") pod \"authorino-755b8f8b87-6cqsw\" (UID: \"65aa5bd7-3f91-4491-8a85-c686955e8e85\") " pod="kuadrant-system/authorino-755b8f8b87-6cqsw" Jul 13 22:08:16.608923 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:08:16.608897 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2wsx\" (UniqueName: \"kubernetes.io/projected/65aa5bd7-3f91-4491-8a85-c686955e8e85-kube-api-access-q2wsx\") pod \"authorino-755b8f8b87-6cqsw\" (UID: \"65aa5bd7-3f91-4491-8a85-c686955e8e85\") " pod="kuadrant-system/authorino-755b8f8b87-6cqsw" Jul 13 22:08:16.657904 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:08:16.657871 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-755b8f8b87-6cqsw" Jul 13 22:08:16.793509 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:08:16.793482 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-755b8f8b87-6cqsw"] Jul 13 22:08:16.799753 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:08:16.799729 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod65aa5bd7_3f91_4491_8a85_c686955e8e85.slice/crio-63f307b1eff6aa84683b7858ba2c139f6ae0a917f8874ba46d39e186d0adea16 WatchSource:0}: Error finding container 63f307b1eff6aa84683b7858ba2c139f6ae0a917f8874ba46d39e186d0adea16: Status 404 returned error can't find the container with id 63f307b1eff6aa84683b7858ba2c139f6ae0a917f8874ba46d39e186d0adea16 Jul 13 22:08:16.815979 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:08:16.815943 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-kwcfn" event={"ID":"a77e7c2c-afde-48e0-9f74-8c2af82c8e85","Type":"ContainerStarted","Data":"d4a7b99820f33fe86cb6d31f802330b9aa082c8f9e6a9fbdc7d7ca94eadd3b1c"} Jul 13 22:08:16.817343 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:08:16.817326 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-64bf5445dc-kh6bg" Jul 13 22:08:16.817434 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:08:16.817347 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-755b8f8b87-6cqsw" event={"ID":"65aa5bd7-3f91-4491-8a85-c686955e8e85","Type":"ContainerStarted","Data":"63f307b1eff6aa84683b7858ba2c139f6ae0a917f8874ba46d39e186d0adea16"} Jul 13 22:08:16.823406 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:08:16.823387 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-64bf5445dc-kh6bg" Jul 13 22:08:16.904333 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:08:16.904300 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6q272\" (UniqueName: \"kubernetes.io/projected/ceb42c6b-015a-4c8c-a449-c630c5014c21-kube-api-access-6q272\") pod \"ceb42c6b-015a-4c8c-a449-c630c5014c21\" (UID: \"ceb42c6b-015a-4c8c-a449-c630c5014c21\") " Jul 13 22:08:16.906874 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:08:16.906850 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceb42c6b-015a-4c8c-a449-c630c5014c21-kube-api-access-6q272" (OuterVolumeSpecName: "kube-api-access-6q272") pod "ceb42c6b-015a-4c8c-a449-c630c5014c21" (UID: "ceb42c6b-015a-4c8c-a449-c630c5014c21"). InnerVolumeSpecName "kube-api-access-6q272". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:08:17.005612 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:08:17.005579 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6q272\" (UniqueName: \"kubernetes.io/projected/ceb42c6b-015a-4c8c-a449-c630c5014c21-kube-api-access-6q272\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:08:17.824608 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:08:17.824573 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-755b8f8b87-6cqsw" event={"ID":"65aa5bd7-3f91-4491-8a85-c686955e8e85","Type":"ContainerStarted","Data":"2bf757f8c35501431cda264a81e6330d7e84fa82d76ac2d3312cedf6fa7a477b"} Jul 13 22:08:17.826081 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:08:17.826062 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-64bf5445dc-kh6bg" Jul 13 22:08:17.826224 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:08:17.826107 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-kwcfn" event={"ID":"a77e7c2c-afde-48e0-9f74-8c2af82c8e85","Type":"ContainerStarted","Data":"353feaa32fbb216ccef529cd88a8aea4d4b609a62bedccd31477517f7bd0a123"} Jul 13 22:08:17.826224 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:08:17.826056 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-kwcfn" podUID="a77e7c2c-afde-48e0-9f74-8c2af82c8e85" containerName="authorino" containerID="cri-o://353feaa32fbb216ccef529cd88a8aea4d4b609a62bedccd31477517f7bd0a123" gracePeriod=30 Jul 13 22:08:17.841547 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:08:17.841508 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-755b8f8b87-6cqsw" podStartSLOduration=1.549201179 podStartE2EDuration="1.841494915s" totalImagesPullingTime="292.293736ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:08:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:08:17.839258155 +0000 UTC m=+503.833012452" watchObservedRunningTime="2026-07-13 22:08:17.841494915 +0000 UTC m=+503.835249208" Jul 13 22:08:17.864248 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:08:17.864195 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-64bf5445dc-kh6bg"] Jul 13 22:08:17.872903 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:08:17.872880 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-64bf5445dc-kh6bg"] Jul 13 22:08:17.891163 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:08:17.891107 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-kwcfn" podStartSLOduration=2.50078052 podStartE2EDuration="2.891086304s" totalImagesPullingTime="390.305784ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:08:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:08:17.889111597 +0000 UTC m=+503.882865889" watchObservedRunningTime="2026-07-13 22:08:17.891086304 +0000 UTC m=+503.884840598" Jul 13 22:08:18.076069 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:08:18.076017 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-kwcfn" Jul 13 22:08:18.217128 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:08:18.217095 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lr68j\" (UniqueName: \"kubernetes.io/projected/a77e7c2c-afde-48e0-9f74-8c2af82c8e85-kube-api-access-lr68j\") pod \"a77e7c2c-afde-48e0-9f74-8c2af82c8e85\" (UID: \"a77e7c2c-afde-48e0-9f74-8c2af82c8e85\") " Jul 13 22:08:18.219246 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:08:18.219218 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a77e7c2c-afde-48e0-9f74-8c2af82c8e85-kube-api-access-lr68j" (OuterVolumeSpecName: "kube-api-access-lr68j") pod "a77e7c2c-afde-48e0-9f74-8c2af82c8e85" (UID: "a77e7c2c-afde-48e0-9f74-8c2af82c8e85"). InnerVolumeSpecName "kube-api-access-lr68j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:08:18.317787 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:08:18.317750 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lr68j\" (UniqueName: \"kubernetes.io/projected/a77e7c2c-afde-48e0-9f74-8c2af82c8e85-kube-api-access-lr68j\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:08:18.559322 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:08:18.559284 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ceb42c6b-015a-4c8c-a449-c630c5014c21" path="/var/lib/kubelet/pods/ceb42c6b-015a-4c8c-a449-c630c5014c21/volumes" Jul 13 22:08:18.832844 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:08:18.832760 2555 generic.go:356] "Generic (PLEG): container finished" podID="a77e7c2c-afde-48e0-9f74-8c2af82c8e85" containerID="353feaa32fbb216ccef529cd88a8aea4d4b609a62bedccd31477517f7bd0a123" exitCode=0 Jul 13 22:08:18.832844 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:08:18.832806 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-kwcfn" Jul 13 22:08:18.833034 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:08:18.832849 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-kwcfn" event={"ID":"a77e7c2c-afde-48e0-9f74-8c2af82c8e85","Type":"ContainerDied","Data":"353feaa32fbb216ccef529cd88a8aea4d4b609a62bedccd31477517f7bd0a123"} Jul 13 22:08:18.833034 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:08:18.832886 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-kwcfn" event={"ID":"a77e7c2c-afde-48e0-9f74-8c2af82c8e85","Type":"ContainerDied","Data":"d4a7b99820f33fe86cb6d31f802330b9aa082c8f9e6a9fbdc7d7ca94eadd3b1c"} Jul 13 22:08:18.833034 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:08:18.832907 2555 scope.go:117] "RemoveContainer" containerID="353feaa32fbb216ccef529cd88a8aea4d4b609a62bedccd31477517f7bd0a123" Jul 13 22:08:18.841688 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:08:18.841673 2555 scope.go:117] "RemoveContainer" containerID="353feaa32fbb216ccef529cd88a8aea4d4b609a62bedccd31477517f7bd0a123" Jul 13 22:08:18.841953 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:08:18.841934 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"353feaa32fbb216ccef529cd88a8aea4d4b609a62bedccd31477517f7bd0a123\": container with ID starting with 353feaa32fbb216ccef529cd88a8aea4d4b609a62bedccd31477517f7bd0a123 not found: ID does not exist" containerID="353feaa32fbb216ccef529cd88a8aea4d4b609a62bedccd31477517f7bd0a123" Jul 13 22:08:18.841997 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:08:18.841962 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"353feaa32fbb216ccef529cd88a8aea4d4b609a62bedccd31477517f7bd0a123"} err="failed to get container status \"353feaa32fbb216ccef529cd88a8aea4d4b609a62bedccd31477517f7bd0a123\": rpc error: code = NotFound desc = could not find container \"353feaa32fbb216ccef529cd88a8aea4d4b609a62bedccd31477517f7bd0a123\": container with ID starting with 353feaa32fbb216ccef529cd88a8aea4d4b609a62bedccd31477517f7bd0a123 not found: ID does not exist" Jul 13 22:08:18.848676 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:08:18.848658 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-kwcfn"] Jul 13 22:08:18.852684 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:08:18.852666 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-kwcfn"] Jul 13 22:08:20.559603 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:08:20.559570 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a77e7c2c-afde-48e0-9f74-8c2af82c8e85" path="/var/lib/kubelet/pods/a77e7c2c-afde-48e0-9f74-8c2af82c8e85/volumes" Jul 13 22:09:07.079451 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:09:07.079415 2555 generic.go:356] "Generic (PLEG): container finished" podID="504e7d67-a82c-4653-bb50-a64a7a240eb5" containerID="c65cea25b749181159f7ae7a1b8ec11067abaa6fb8d9fe79853b86e758f56209" exitCode=0 Jul 13 22:09:07.079869 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:09:07.079492 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jhf2p" event={"ID":"504e7d67-a82c-4653-bb50-a64a7a240eb5","Type":"ContainerDied","Data":"c65cea25b749181159f7ae7a1b8ec11067abaa6fb8d9fe79853b86e758f56209"} Jul 13 22:09:07.079939 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:09:07.079924 2555 scope.go:117] "RemoveContainer" containerID="c65cea25b749181159f7ae7a1b8ec11067abaa6fb8d9fe79853b86e758f56209" Jul 13 22:09:07.080892 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:09:07.080864 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:09:07.080950 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:09:07.080937 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:09:07.080984 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:09:07.080974 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:09:08.087091 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:09:08.087059 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jhf2p" event={"ID":"504e7d67-a82c-4653-bb50-a64a7a240eb5","Type":"ContainerStarted","Data":"3a380b0352f4ac9709ab34f710222d295047f230ef87d8da164e595eec5a69e8"} Jul 13 22:09:08.219436 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:09:08.219390 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jhf2p" Jul 13 22:09:08.220440 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:09:08.220414 2555 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-jhf2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 13 22:09:08.220563 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:09:08.220464 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jhf2p" podUID="504e7d67-a82c-4653-bb50-a64a7a240eb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 13 22:09:09.219130 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:09:09.219097 2555 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-jhf2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 13 22:09:09.219575 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:09:09.219153 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jhf2p" podUID="504e7d67-a82c-4653-bb50-a64a7a240eb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 13 22:09:10.220143 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:09:10.220105 2555 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-jhf2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 13 22:09:10.220558 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:09:10.220167 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jhf2p" podUID="504e7d67-a82c-4653-bb50-a64a7a240eb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 13 22:09:10.612983 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:09:10.612887 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jhf2p" Jul 13 22:09:11.219424 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:09:11.219388 2555 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-jhf2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 13 22:09:11.219662 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:09:11.219641 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jhf2p" podUID="504e7d67-a82c-4653-bb50-a64a7a240eb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 13 22:09:12.223276 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:09:12.223247 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jhf2p" Jul 13 22:09:12.224160 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:09:12.224144 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-jhf2p" Jul 13 22:09:13.273881 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:09:13.273842 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vgvcv"] Jul 13 22:09:13.274276 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:09:13.274261 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a77e7c2c-afde-48e0-9f74-8c2af82c8e85" containerName="authorino" Jul 13 22:09:13.274331 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:09:13.274278 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a77e7c2c-afde-48e0-9f74-8c2af82c8e85" containerName="authorino" Jul 13 22:09:13.274366 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:09:13.274342 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a77e7c2c-afde-48e0-9f74-8c2af82c8e85" containerName="authorino" Jul 13 22:09:13.277086 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:09:13.277068 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vgvcv" Jul 13 22:09:13.284378 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:09:13.284352 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 13 22:09:13.284499 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:09:13.284449 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 13 22:09:13.285427 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:09:13.285410 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 13 22:09:13.285511 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:09:13.285410 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-gfrsn\"" Jul 13 22:09:13.298624 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:09:13.298599 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vgvcv"] Jul 13 22:09:13.385220 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:09:13.385175 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49c24049-acad-430c-8476-44c489ad4b32-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vgvcv\" (UID: \"49c24049-acad-430c-8476-44c489ad4b32\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vgvcv" Jul 13 22:09:13.385408 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:09:13.385235 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49c24049-acad-430c-8476-44c489ad4b32-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vgvcv\" (UID: \"49c24049-acad-430c-8476-44c489ad4b32\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vgvcv" Jul 13 22:09:13.385408 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:09:13.385297 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49c24049-acad-430c-8476-44c489ad4b32-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vgvcv\" (UID: \"49c24049-acad-430c-8476-44c489ad4b32\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vgvcv" Jul 13 22:09:13.385408 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:09:13.385324 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49c24049-acad-430c-8476-44c489ad4b32-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vgvcv\" (UID: \"49c24049-acad-430c-8476-44c489ad4b32\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vgvcv" Jul 13 22:09:13.385558 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:09:13.385401 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49c24049-acad-430c-8476-44c489ad4b32-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vgvcv\" (UID: \"49c24049-acad-430c-8476-44c489ad4b32\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vgvcv" Jul 13 22:09:13.385558 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:09:13.385440 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgflm\" (UniqueName: \"kubernetes.io/projected/49c24049-acad-430c-8476-44c489ad4b32-kube-api-access-bgflm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vgvcv\" (UID: \"49c24049-acad-430c-8476-44c489ad4b32\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vgvcv" Jul 13 22:09:13.486692 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:09:13.486655 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49c24049-acad-430c-8476-44c489ad4b32-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vgvcv\" (UID: \"49c24049-acad-430c-8476-44c489ad4b32\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vgvcv" Jul 13 22:09:13.486692 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:09:13.486696 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49c24049-acad-430c-8476-44c489ad4b32-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vgvcv\" (UID: \"49c24049-acad-430c-8476-44c489ad4b32\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vgvcv" Jul 13 22:09:13.486928 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:09:13.486725 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49c24049-acad-430c-8476-44c489ad4b32-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vgvcv\" (UID: \"49c24049-acad-430c-8476-44c489ad4b32\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vgvcv" Jul 13 22:09:13.486928 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:09:13.486750 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgflm\" (UniqueName: \"kubernetes.io/projected/49c24049-acad-430c-8476-44c489ad4b32-kube-api-access-bgflm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vgvcv\" (UID: \"49c24049-acad-430c-8476-44c489ad4b32\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vgvcv" Jul 13 22:09:13.486928 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:09:13.486850 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49c24049-acad-430c-8476-44c489ad4b32-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vgvcv\" (UID: \"49c24049-acad-430c-8476-44c489ad4b32\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vgvcv" Jul 13 22:09:13.486928 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:09:13.486879 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49c24049-acad-430c-8476-44c489ad4b32-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vgvcv\" (UID: \"49c24049-acad-430c-8476-44c489ad4b32\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vgvcv" Jul 13 22:09:13.487149 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:09:13.487121 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49c24049-acad-430c-8476-44c489ad4b32-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vgvcv\" (UID: \"49c24049-acad-430c-8476-44c489ad4b32\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vgvcv" Jul 13 22:09:13.487228 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:09:13.487150 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49c24049-acad-430c-8476-44c489ad4b32-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vgvcv\" (UID: \"49c24049-acad-430c-8476-44c489ad4b32\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vgvcv" Jul 13 22:09:13.487352 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:09:13.487235 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49c24049-acad-430c-8476-44c489ad4b32-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vgvcv\" (UID: \"49c24049-acad-430c-8476-44c489ad4b32\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vgvcv" Jul 13 22:09:13.489130 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:09:13.489106 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49c24049-acad-430c-8476-44c489ad4b32-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vgvcv\" (UID: \"49c24049-acad-430c-8476-44c489ad4b32\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vgvcv" Jul 13 22:09:13.489424 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:09:13.489405 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49c24049-acad-430c-8476-44c489ad4b32-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vgvcv\" (UID: \"49c24049-acad-430c-8476-44c489ad4b32\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vgvcv" Jul 13 22:09:13.498400 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:09:13.498378 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgflm\" (UniqueName: \"kubernetes.io/projected/49c24049-acad-430c-8476-44c489ad4b32-kube-api-access-bgflm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vgvcv\" (UID: \"49c24049-acad-430c-8476-44c489ad4b32\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vgvcv" Jul 13 22:09:13.587683 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:09:13.587592 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vgvcv" Jul 13 22:09:13.739342 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:09:13.733298 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vgvcv"] Jul 13 22:09:14.129084 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:09:14.129044 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vgvcv" event={"ID":"49c24049-acad-430c-8476-44c489ad4b32","Type":"ContainerStarted","Data":"0cbb9507313f782031129043044ad4b013153a1141aff4975d8245c8daecc525"} Jul 13 22:09:15.135931 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:09:15.135883 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vgvcv" event={"ID":"49c24049-acad-430c-8476-44c489ad4b32","Type":"ContainerStarted","Data":"b77b34347ea1f2bb3c91d69a49e4dfa4b9a39292a8fc0ca0937b0f8059a03da2"} Jul 13 22:09:15.156610 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:09:15.156562 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vgvcv" podStartSLOduration=0.971235046 podStartE2EDuration="2.156546249s" totalImagesPullingTime="1.185311203s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:09:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:09:15.156049925 +0000 UTC m=+561.149804238" watchObservedRunningTime="2026-07-13 22:09:15.156546249 +0000 UTC m=+561.150300542" Jul 13 22:09:23.588575 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:09:23.588537 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vgvcv" Jul 13 22:09:23.588976 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:09:23.588607 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vgvcv" Jul 13 22:09:23.601100 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:09:23.601072 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vgvcv" Jul 13 22:09:24.193476 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:09:24.193446 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vgvcv" Jul 13 22:11:13.935582 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:11:13.935547 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-64d6554466-4jltm"] Jul 13 22:11:13.938903 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:11:13.938884 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-64d6554466-4jltm" Jul 13 22:11:13.951014 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:11:13.950993 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-64d6554466-4jltm"] Jul 13 22:11:14.129774 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:11:14.129739 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cfc6882c-943b-49f5-843d-490b9e6e075b-tls-cert\") pod \"authorino-64d6554466-4jltm\" (UID: \"cfc6882c-943b-49f5-843d-490b9e6e075b\") " pod="kuadrant-system/authorino-64d6554466-4jltm" Jul 13 22:11:14.129954 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:11:14.129794 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkbtr\" (UniqueName: \"kubernetes.io/projected/cfc6882c-943b-49f5-843d-490b9e6e075b-kube-api-access-mkbtr\") pod \"authorino-64d6554466-4jltm\" (UID: \"cfc6882c-943b-49f5-843d-490b9e6e075b\") " pod="kuadrant-system/authorino-64d6554466-4jltm" Jul 13 22:11:14.230654 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:11:14.230553 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cfc6882c-943b-49f5-843d-490b9e6e075b-tls-cert\") pod \"authorino-64d6554466-4jltm\" (UID: \"cfc6882c-943b-49f5-843d-490b9e6e075b\") " pod="kuadrant-system/authorino-64d6554466-4jltm" Jul 13 22:11:14.230654 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:11:14.230609 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkbtr\" (UniqueName: \"kubernetes.io/projected/cfc6882c-943b-49f5-843d-490b9e6e075b-kube-api-access-mkbtr\") pod \"authorino-64d6554466-4jltm\" (UID: \"cfc6882c-943b-49f5-843d-490b9e6e075b\") " pod="kuadrant-system/authorino-64d6554466-4jltm" Jul 13 22:11:14.233276 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:11:14.233249 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cfc6882c-943b-49f5-843d-490b9e6e075b-tls-cert\") pod \"authorino-64d6554466-4jltm\" (UID: \"cfc6882c-943b-49f5-843d-490b9e6e075b\") " pod="kuadrant-system/authorino-64d6554466-4jltm" Jul 13 22:11:14.240428 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:11:14.240403 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkbtr\" (UniqueName: \"kubernetes.io/projected/cfc6882c-943b-49f5-843d-490b9e6e075b-kube-api-access-mkbtr\") pod \"authorino-64d6554466-4jltm\" (UID: \"cfc6882c-943b-49f5-843d-490b9e6e075b\") " pod="kuadrant-system/authorino-64d6554466-4jltm" Jul 13 22:11:14.247138 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:11:14.247113 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-64d6554466-4jltm" Jul 13 22:11:14.405535 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:11:14.405512 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-64d6554466-4jltm"] Jul 13 22:11:14.407285 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:11:14.407262 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcfc6882c_943b_49f5_843d_490b9e6e075b.slice/crio-b772c72e9a2b5407018667beced91523e82744e2b1bb78c4de52a19fd04a33f8 WatchSource:0}: Error finding container b772c72e9a2b5407018667beced91523e82744e2b1bb78c4de52a19fd04a33f8: Status 404 returned error can't find the container with id b772c72e9a2b5407018667beced91523e82744e2b1bb78c4de52a19fd04a33f8 Jul 13 22:11:14.408631 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:11:14.408613 2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 13 22:11:14.778788 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:11:14.778753 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-64d6554466-4jltm" event={"ID":"cfc6882c-943b-49f5-843d-490b9e6e075b","Type":"ContainerStarted","Data":"b772c72e9a2b5407018667beced91523e82744e2b1bb78c4de52a19fd04a33f8"} Jul 13 22:11:15.785404 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:11:15.785359 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-64d6554466-4jltm" event={"ID":"cfc6882c-943b-49f5-843d-490b9e6e075b","Type":"ContainerStarted","Data":"d47bbb6409dfc2c5ee96aef026d529ea95fed5f4b6d4f372525c14ce7024a930"} Jul 13 22:11:15.808383 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:11:15.808325 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-64d6554466-4jltm" podStartSLOduration=2.283663328 podStartE2EDuration="2.808301597s" totalImagesPullingTime="524.638269ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:11:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:11:15.806517373 +0000 UTC m=+681.800271668" watchObservedRunningTime="2026-07-13 22:11:15.808301597 +0000 UTC m=+681.802055893" Jul 13 22:11:15.880364 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:11:15.880319 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-755b8f8b87-6cqsw"] Jul 13 22:11:15.880660 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:11:15.880627 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-755b8f8b87-6cqsw" podUID="65aa5bd7-3f91-4491-8a85-c686955e8e85" containerName="authorino" containerID="cri-o://2bf757f8c35501431cda264a81e6330d7e84fa82d76ac2d3312cedf6fa7a477b" gracePeriod=30 Jul 13 22:11:16.144158 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:11:16.144130 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-755b8f8b87-6cqsw" Jul 13 22:11:16.248773 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:11:16.248736 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/65aa5bd7-3f91-4491-8a85-c686955e8e85-tls-cert\") pod \"65aa5bd7-3f91-4491-8a85-c686955e8e85\" (UID: \"65aa5bd7-3f91-4491-8a85-c686955e8e85\") " Jul 13 22:11:16.248773 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:11:16.248782 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q2wsx\" (UniqueName: \"kubernetes.io/projected/65aa5bd7-3f91-4491-8a85-c686955e8e85-kube-api-access-q2wsx\") pod \"65aa5bd7-3f91-4491-8a85-c686955e8e85\" (UID: \"65aa5bd7-3f91-4491-8a85-c686955e8e85\") " Jul 13 22:11:16.251028 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:11:16.250988 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65aa5bd7-3f91-4491-8a85-c686955e8e85-kube-api-access-q2wsx" (OuterVolumeSpecName: "kube-api-access-q2wsx") pod "65aa5bd7-3f91-4491-8a85-c686955e8e85" (UID: "65aa5bd7-3f91-4491-8a85-c686955e8e85"). InnerVolumeSpecName "kube-api-access-q2wsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:11:16.259748 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:11:16.259703 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65aa5bd7-3f91-4491-8a85-c686955e8e85-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "65aa5bd7-3f91-4491-8a85-c686955e8e85" (UID: "65aa5bd7-3f91-4491-8a85-c686955e8e85"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 22:11:16.350465 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:11:16.350378 2555 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/65aa5bd7-3f91-4491-8a85-c686955e8e85-tls-cert\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:11:16.350465 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:11:16.350410 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2wsx\" (UniqueName: \"kubernetes.io/projected/65aa5bd7-3f91-4491-8a85-c686955e8e85-kube-api-access-q2wsx\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:11:16.792510 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:11:16.792470 2555 generic.go:356] "Generic (PLEG): container finished" podID="65aa5bd7-3f91-4491-8a85-c686955e8e85" containerID="2bf757f8c35501431cda264a81e6330d7e84fa82d76ac2d3312cedf6fa7a477b" exitCode=0 Jul 13 22:11:16.792976 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:11:16.792567 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-755b8f8b87-6cqsw" Jul 13 22:11:16.792976 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:11:16.792593 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-755b8f8b87-6cqsw" event={"ID":"65aa5bd7-3f91-4491-8a85-c686955e8e85","Type":"ContainerDied","Data":"2bf757f8c35501431cda264a81e6330d7e84fa82d76ac2d3312cedf6fa7a477b"} Jul 13 22:11:16.792976 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:11:16.792641 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-755b8f8b87-6cqsw" event={"ID":"65aa5bd7-3f91-4491-8a85-c686955e8e85","Type":"ContainerDied","Data":"63f307b1eff6aa84683b7858ba2c139f6ae0a917f8874ba46d39e186d0adea16"} Jul 13 22:11:16.792976 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:11:16.792659 2555 scope.go:117] "RemoveContainer" containerID="2bf757f8c35501431cda264a81e6330d7e84fa82d76ac2d3312cedf6fa7a477b" Jul 13 22:11:16.803694 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:11:16.803677 2555 scope.go:117] "RemoveContainer" containerID="2bf757f8c35501431cda264a81e6330d7e84fa82d76ac2d3312cedf6fa7a477b" Jul 13 22:11:16.804012 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:11:16.803991 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bf757f8c35501431cda264a81e6330d7e84fa82d76ac2d3312cedf6fa7a477b\": container with ID starting with 2bf757f8c35501431cda264a81e6330d7e84fa82d76ac2d3312cedf6fa7a477b not found: ID does not exist" containerID="2bf757f8c35501431cda264a81e6330d7e84fa82d76ac2d3312cedf6fa7a477b" Jul 13 22:11:16.804064 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:11:16.804022 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bf757f8c35501431cda264a81e6330d7e84fa82d76ac2d3312cedf6fa7a477b"} err="failed to get container status \"2bf757f8c35501431cda264a81e6330d7e84fa82d76ac2d3312cedf6fa7a477b\": rpc error: code = NotFound desc = could not find container \"2bf757f8c35501431cda264a81e6330d7e84fa82d76ac2d3312cedf6fa7a477b\": container with ID starting with 2bf757f8c35501431cda264a81e6330d7e84fa82d76ac2d3312cedf6fa7a477b not found: ID does not exist" Jul 13 22:11:16.832187 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:11:16.832154 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-755b8f8b87-6cqsw"] Jul 13 22:11:16.854795 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:11:16.854765 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-755b8f8b87-6cqsw"] Jul 13 22:11:18.559544 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:11:18.559515 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65aa5bd7-3f91-4491-8a85-c686955e8e85" path="/var/lib/kubelet/pods/65aa5bd7-3f91-4491-8a85-c686955e8e85/volumes" Jul 13 22:15:00.149743 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:15:00.149704 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29733015-5wsdj"] Jul 13 22:15:00.150172 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:15:00.150073 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65aa5bd7-3f91-4491-8a85-c686955e8e85" containerName="authorino" Jul 13 22:15:00.150172 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:15:00.150084 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="65aa5bd7-3f91-4491-8a85-c686955e8e85" containerName="authorino" Jul 13 22:15:00.150172 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:15:00.150155 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="65aa5bd7-3f91-4491-8a85-c686955e8e85" containerName="authorino" Jul 13 22:15:00.153251 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:15:00.153234 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733015-5wsdj" Jul 13 22:15:00.155780 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:15:00.155760 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k5z5d\"" Jul 13 22:15:00.162487 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:15:00.162468 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733015-5wsdj" Jul 13 22:15:00.175467 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:15:00.175435 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29733015-5wsdj"] Jul 13 22:15:00.310305 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:15:00.310279 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29733015-5wsdj"] Jul 13 22:15:00.312444 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:15:00.312417 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd8a23da_b88e_4e5f_8d41_761484e073d6.slice/crio-52cf298e3f9e9a7eb236bc1269e0b6e6accdffe5422bf91d257efca6693196e7 WatchSource:0}: Error finding container 52cf298e3f9e9a7eb236bc1269e0b6e6accdffe5422bf91d257efca6693196e7: Status 404 returned error can't find the container with id 52cf298e3f9e9a7eb236bc1269e0b6e6accdffe5422bf91d257efca6693196e7 Jul 13 22:15:00.990666 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:15:00.990631 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733015-5wsdj" event={"ID":"fd8a23da-b88e-4e5f-8d41-761484e073d6","Type":"ContainerStarted","Data":"52cf298e3f9e9a7eb236bc1269e0b6e6accdffe5422bf91d257efca6693196e7"} Jul 13 22:15:03.003291 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:15:03.003247 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733015-5wsdj" event={"ID":"fd8a23da-b88e-4e5f-8d41-761484e073d6","Type":"ContainerStarted","Data":"6ca172d21970bf277e74180db687999714a05e6946ef445c87891c0c26dbc487"} Jul 13 22:15:03.022146 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:15:03.022081 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733015-5wsdj" podStartSLOduration=0.969347971 podStartE2EDuration="3.022065961s" totalImagesPullingTime="2.05271799s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:15:03.019722838 +0000 UTC m=+909.013477168" watchObservedRunningTime="2026-07-13 22:15:03.022065961 +0000 UTC m=+909.015820254" Jul 13 22:15:04.011794 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:15:04.011749 2555 generic.go:356] "Generic (PLEG): container finished" podID="fd8a23da-b88e-4e5f-8d41-761484e073d6" containerID="6ca172d21970bf277e74180db687999714a05e6946ef445c87891c0c26dbc487" exitCode=0 Jul 13 22:15:04.012274 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:15:04.011819 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733015-5wsdj" event={"ID":"fd8a23da-b88e-4e5f-8d41-761484e073d6","Type":"ContainerDied","Data":"6ca172d21970bf277e74180db687999714a05e6946ef445c87891c0c26dbc487"} Jul 13 22:15:05.153856 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:15:05.153829 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733015-5wsdj" Jul 13 22:15:06.023599 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:15:06.023562 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733015-5wsdj" Jul 13 22:15:06.023599 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:15:06.023595 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733015-5wsdj" event={"ID":"fd8a23da-b88e-4e5f-8d41-761484e073d6","Type":"ContainerDied","Data":"52cf298e3f9e9a7eb236bc1269e0b6e6accdffe5422bf91d257efca6693196e7"} Jul 13 22:15:06.023811 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:15:06.023643 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52cf298e3f9e9a7eb236bc1269e0b6e6accdffe5422bf91d257efca6693196e7" Jul 13 22:17:05.998499 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:17:05.998465 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5wxmf"] Jul 13 22:17:05.999065 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:17:05.998697 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5wxmf" podUID="23dbc2e4-e4d6-446c-93fe-4212b860f025" containerName="manager" containerID="cri-o://ee10a5fb4e6dc2d5c38b88a7cd38ad8ce7df7176dae056bd8cef78fe02198bf7" gracePeriod=10 Jul 13 22:17:08.957752 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:17:08.957722 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5wxmf" Jul 13 22:17:09.054122 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:17:09.054037 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/23dbc2e4-e4d6-446c-93fe-4212b860f025-extensions-socket-volume\") pod \"23dbc2e4-e4d6-446c-93fe-4212b860f025\" (UID: \"23dbc2e4-e4d6-446c-93fe-4212b860f025\") " Jul 13 22:17:09.054122 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:17:09.054086 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wtgmm\" (UniqueName: \"kubernetes.io/projected/23dbc2e4-e4d6-446c-93fe-4212b860f025-kube-api-access-wtgmm\") pod \"23dbc2e4-e4d6-446c-93fe-4212b860f025\" (UID: \"23dbc2e4-e4d6-446c-93fe-4212b860f025\") " Jul 13 22:17:09.054402 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:17:09.054377 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23dbc2e4-e4d6-446c-93fe-4212b860f025-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "23dbc2e4-e4d6-446c-93fe-4212b860f025" (UID: "23dbc2e4-e4d6-446c-93fe-4212b860f025"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:17:09.056211 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:17:09.056175 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23dbc2e4-e4d6-446c-93fe-4212b860f025-kube-api-access-wtgmm" (OuterVolumeSpecName: "kube-api-access-wtgmm") pod "23dbc2e4-e4d6-446c-93fe-4212b860f025" (UID: "23dbc2e4-e4d6-446c-93fe-4212b860f025"). InnerVolumeSpecName "kube-api-access-wtgmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:17:09.155145 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:17:09.155115 2555 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/23dbc2e4-e4d6-446c-93fe-4212b860f025-extensions-socket-volume\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:17:09.155145 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:17:09.155140 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtgmm\" (UniqueName: \"kubernetes.io/projected/23dbc2e4-e4d6-446c-93fe-4212b860f025-kube-api-access-wtgmm\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:17:09.711268 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:17:09.711195 2555 generic.go:356] "Generic (PLEG): container finished" podID="23dbc2e4-e4d6-446c-93fe-4212b860f025" containerID="ee10a5fb4e6dc2d5c38b88a7cd38ad8ce7df7176dae056bd8cef78fe02198bf7" exitCode=0 Jul 13 22:17:09.711268 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:17:09.711256 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5wxmf" Jul 13 22:17:09.711508 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:17:09.711240 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5wxmf" event={"ID":"23dbc2e4-e4d6-446c-93fe-4212b860f025","Type":"ContainerDied","Data":"ee10a5fb4e6dc2d5c38b88a7cd38ad8ce7df7176dae056bd8cef78fe02198bf7"} Jul 13 22:17:09.711508 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:17:09.711364 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5wxmf" event={"ID":"23dbc2e4-e4d6-446c-93fe-4212b860f025","Type":"ContainerDied","Data":"14c0bfe12e63a3d59a103fbafc60fd5487e9fca1c08a7c463b58eb31e5afc0f4"} Jul 13 22:17:09.711508 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:17:09.711386 2555 scope.go:117] "RemoveContainer" containerID="ee10a5fb4e6dc2d5c38b88a7cd38ad8ce7df7176dae056bd8cef78fe02198bf7" Jul 13 22:17:09.726713 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:17:09.726694 2555 scope.go:117] "RemoveContainer" containerID="ee10a5fb4e6dc2d5c38b88a7cd38ad8ce7df7176dae056bd8cef78fe02198bf7" Jul 13 22:17:09.726957 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:17:09.726927 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee10a5fb4e6dc2d5c38b88a7cd38ad8ce7df7176dae056bd8cef78fe02198bf7\": container with ID starting with ee10a5fb4e6dc2d5c38b88a7cd38ad8ce7df7176dae056bd8cef78fe02198bf7 not found: ID does not exist" containerID="ee10a5fb4e6dc2d5c38b88a7cd38ad8ce7df7176dae056bd8cef78fe02198bf7" Jul 13 22:17:09.727057 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:17:09.726966 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee10a5fb4e6dc2d5c38b88a7cd38ad8ce7df7176dae056bd8cef78fe02198bf7"} err="failed to get container status \"ee10a5fb4e6dc2d5c38b88a7cd38ad8ce7df7176dae056bd8cef78fe02198bf7\": rpc error: code = NotFound desc = could not find container \"ee10a5fb4e6dc2d5c38b88a7cd38ad8ce7df7176dae056bd8cef78fe02198bf7\": container with ID starting with ee10a5fb4e6dc2d5c38b88a7cd38ad8ce7df7176dae056bd8cef78fe02198bf7 not found: ID does not exist" Jul 13 22:17:09.737599 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:17:09.737579 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5wxmf"] Jul 13 22:17:09.741173 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:17:09.741155 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5wxmf"] Jul 13 22:17:10.559165 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:17:10.559133 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23dbc2e4-e4d6-446c-93fe-4212b860f025" path="/var/lib/kubelet/pods/23dbc2e4-e4d6-446c-93fe-4212b860f025/volumes" Jul 13 22:18:07.213385 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:18:07.213348 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rgbkf"] Jul 13 22:18:07.213790 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:18:07.213689 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23dbc2e4-e4d6-446c-93fe-4212b860f025" containerName="manager" Jul 13 22:18:07.213790 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:18:07.213698 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="23dbc2e4-e4d6-446c-93fe-4212b860f025" containerName="manager" Jul 13 22:18:07.213790 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:18:07.213713 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd8a23da-b88e-4e5f-8d41-761484e073d6" containerName="cleanup" Jul 13 22:18:07.213790 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:18:07.213719 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd8a23da-b88e-4e5f-8d41-761484e073d6" containerName="cleanup" Jul 13 22:18:07.213790 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:18:07.213781 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="23dbc2e4-e4d6-446c-93fe-4212b860f025" containerName="manager" Jul 13 22:18:07.213790 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:18:07.213791 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd8a23da-b88e-4e5f-8d41-761484e073d6" containerName="cleanup" Jul 13 22:18:07.217837 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:18:07.217819 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rgbkf" Jul 13 22:18:07.220694 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:18:07.220675 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-9jb6z\"" Jul 13 22:18:07.236936 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:18:07.236911 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rgbkf"] Jul 13 22:18:07.327315 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:18:07.327271 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f3d9361d-cb18-499e-9a90-484ea2d9f996-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-rgbkf\" (UID: \"f3d9361d-cb18-499e-9a90-484ea2d9f996\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rgbkf" Jul 13 22:18:07.327497 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:18:07.327337 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27flc\" (UniqueName: \"kubernetes.io/projected/f3d9361d-cb18-499e-9a90-484ea2d9f996-kube-api-access-27flc\") pod \"kuadrant-operator-controller-manager-86878c769f-rgbkf\" (UID: \"f3d9361d-cb18-499e-9a90-484ea2d9f996\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rgbkf" Jul 13 22:18:07.428802 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:18:07.428761 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f3d9361d-cb18-499e-9a90-484ea2d9f996-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-rgbkf\" (UID: \"f3d9361d-cb18-499e-9a90-484ea2d9f996\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rgbkf" Jul 13 22:18:07.428962 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:18:07.428816 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27flc\" (UniqueName: \"kubernetes.io/projected/f3d9361d-cb18-499e-9a90-484ea2d9f996-kube-api-access-27flc\") pod \"kuadrant-operator-controller-manager-86878c769f-rgbkf\" (UID: \"f3d9361d-cb18-499e-9a90-484ea2d9f996\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rgbkf" Jul 13 22:18:07.429186 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:18:07.429165 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f3d9361d-cb18-499e-9a90-484ea2d9f996-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-rgbkf\" (UID: \"f3d9361d-cb18-499e-9a90-484ea2d9f996\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rgbkf" Jul 13 22:18:07.437926 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:18:07.437905 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27flc\" (UniqueName: \"kubernetes.io/projected/f3d9361d-cb18-499e-9a90-484ea2d9f996-kube-api-access-27flc\") pod \"kuadrant-operator-controller-manager-86878c769f-rgbkf\" (UID: \"f3d9361d-cb18-499e-9a90-484ea2d9f996\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rgbkf" Jul 13 22:18:07.528441 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:18:07.528352 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rgbkf" Jul 13 22:18:07.675819 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:18:07.675795 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rgbkf"] Jul 13 22:18:07.678547 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:18:07.678515 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3d9361d_cb18_499e_9a90_484ea2d9f996.slice/crio-4e9cf8e089cd68a2f88818ed22eb034eed3d67d59d166c8343646346a784b173 WatchSource:0}: Error finding container 4e9cf8e089cd68a2f88818ed22eb034eed3d67d59d166c8343646346a784b173: Status 404 returned error can't find the container with id 4e9cf8e089cd68a2f88818ed22eb034eed3d67d59d166c8343646346a784b173 Jul 13 22:18:08.006380 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:18:08.006344 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rgbkf" event={"ID":"f3d9361d-cb18-499e-9a90-484ea2d9f996","Type":"ContainerStarted","Data":"9ddd12a490875433a28ed8a74917f9e9c83305c0b609101bc547b7bd96fffd24"} Jul 13 22:18:08.006562 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:18:08.006401 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rgbkf" event={"ID":"f3d9361d-cb18-499e-9a90-484ea2d9f996","Type":"ContainerStarted","Data":"4e9cf8e089cd68a2f88818ed22eb034eed3d67d59d166c8343646346a784b173"} Jul 13 22:18:08.006562 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:18:08.006429 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rgbkf" Jul 13 22:18:08.030477 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:18:08.030428 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rgbkf" podStartSLOduration=1.030413555 podStartE2EDuration="1.030413555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:18:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:18:08.027696718 +0000 UTC m=+1094.021451035" watchObservedRunningTime="2026-07-13 22:18:08.030413555 +0000 UTC m=+1094.024167848" Jul 13 22:18:19.013672 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:18:19.013642 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rgbkf" Jul 13 22:24:23.094494 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:23.094366 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2"] Jul 13 22:24:23.101008 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:23.100982 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2" Jul 13 22:24:23.104330 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:23.104308 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-d7fae36e-openshift-default-dockercfg-ds59r\"" Jul 13 22:24:23.122989 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:23.122966 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2"] Jul 13 22:24:23.204703 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:23.204663 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c9258fe-311f-403d-91f4-5f0708369da4-istiod-ca-cert\") pod \"e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2\" (UID: \"7c9258fe-311f-403d-91f4-5f0708369da4\") " pod="openshift-ingress/e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2" Jul 13 22:24:23.204703 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:23.204712 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c9258fe-311f-403d-91f4-5f0708369da4-istio-envoy\") pod \"e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2\" (UID: \"7c9258fe-311f-403d-91f4-5f0708369da4\") " pod="openshift-ingress/e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2" Jul 13 22:24:23.204974 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:23.204823 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c9258fe-311f-403d-91f4-5f0708369da4-workload-socket\") pod \"e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2\" (UID: \"7c9258fe-311f-403d-91f4-5f0708369da4\") " pod="openshift-ingress/e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2" Jul 13 22:24:23.204974 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:23.204870 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c9258fe-311f-403d-91f4-5f0708369da4-istio-podinfo\") pod \"e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2\" (UID: \"7c9258fe-311f-403d-91f4-5f0708369da4\") " pod="openshift-ingress/e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2" Jul 13 22:24:23.204974 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:23.204929 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c9258fe-311f-403d-91f4-5f0708369da4-credential-socket\") pod \"e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2\" (UID: \"7c9258fe-311f-403d-91f4-5f0708369da4\") " pod="openshift-ingress/e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2" Jul 13 22:24:23.204974 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:23.204958 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qhzr\" (UniqueName: \"kubernetes.io/projected/7c9258fe-311f-403d-91f4-5f0708369da4-kube-api-access-7qhzr\") pod \"e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2\" (UID: \"7c9258fe-311f-403d-91f4-5f0708369da4\") " pod="openshift-ingress/e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2" Jul 13 22:24:23.205177 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:23.205011 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c9258fe-311f-403d-91f4-5f0708369da4-istio-token\") pod \"e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2\" (UID: \"7c9258fe-311f-403d-91f4-5f0708369da4\") " pod="openshift-ingress/e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2" Jul 13 22:24:23.205177 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:23.205040 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c9258fe-311f-403d-91f4-5f0708369da4-istio-data\") pod \"e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2\" (UID: \"7c9258fe-311f-403d-91f4-5f0708369da4\") " pod="openshift-ingress/e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2" Jul 13 22:24:23.205177 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:23.205072 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c9258fe-311f-403d-91f4-5f0708369da4-workload-certs\") pod \"e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2\" (UID: \"7c9258fe-311f-403d-91f4-5f0708369da4\") " pod="openshift-ingress/e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2" Jul 13 22:24:23.306318 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:23.306282 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c9258fe-311f-403d-91f4-5f0708369da4-workload-socket\") pod \"e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2\" (UID: \"7c9258fe-311f-403d-91f4-5f0708369da4\") " pod="openshift-ingress/e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2" Jul 13 22:24:23.306318 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:23.306324 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c9258fe-311f-403d-91f4-5f0708369da4-istio-podinfo\") pod \"e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2\" (UID: \"7c9258fe-311f-403d-91f4-5f0708369da4\") " pod="openshift-ingress/e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2" Jul 13 22:24:23.306561 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:23.306363 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c9258fe-311f-403d-91f4-5f0708369da4-credential-socket\") pod \"e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2\" (UID: \"7c9258fe-311f-403d-91f4-5f0708369da4\") " pod="openshift-ingress/e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2" Jul 13 22:24:23.306561 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:23.306379 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qhzr\" (UniqueName: \"kubernetes.io/projected/7c9258fe-311f-403d-91f4-5f0708369da4-kube-api-access-7qhzr\") pod \"e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2\" (UID: \"7c9258fe-311f-403d-91f4-5f0708369da4\") " pod="openshift-ingress/e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2" Jul 13 22:24:23.306561 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:23.306408 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c9258fe-311f-403d-91f4-5f0708369da4-istio-token\") pod \"e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2\" (UID: \"7c9258fe-311f-403d-91f4-5f0708369da4\") " pod="openshift-ingress/e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2" Jul 13 22:24:23.306561 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:23.306434 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c9258fe-311f-403d-91f4-5f0708369da4-istio-data\") pod \"e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2\" (UID: \"7c9258fe-311f-403d-91f4-5f0708369da4\") " pod="openshift-ingress/e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2" Jul 13 22:24:23.306561 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:23.306462 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c9258fe-311f-403d-91f4-5f0708369da4-workload-certs\") pod \"e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2\" (UID: \"7c9258fe-311f-403d-91f4-5f0708369da4\") " pod="openshift-ingress/e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2" Jul 13 22:24:23.306561 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:23.306500 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c9258fe-311f-403d-91f4-5f0708369da4-istiod-ca-cert\") pod \"e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2\" (UID: \"7c9258fe-311f-403d-91f4-5f0708369da4\") " pod="openshift-ingress/e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2" Jul 13 22:24:23.306561 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:23.306533 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c9258fe-311f-403d-91f4-5f0708369da4-istio-envoy\") pod \"e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2\" (UID: \"7c9258fe-311f-403d-91f4-5f0708369da4\") " pod="openshift-ingress/e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2" Jul 13 22:24:23.306921 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:23.306807 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c9258fe-311f-403d-91f4-5f0708369da4-istio-data\") pod \"e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2\" (UID: \"7c9258fe-311f-403d-91f4-5f0708369da4\") " pod="openshift-ingress/e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2" Jul 13 22:24:23.306921 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:23.306859 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c9258fe-311f-403d-91f4-5f0708369da4-workload-certs\") pod \"e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2\" (UID: \"7c9258fe-311f-403d-91f4-5f0708369da4\") " pod="openshift-ingress/e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2" Jul 13 22:24:23.307022 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:23.307006 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c9258fe-311f-403d-91f4-5f0708369da4-credential-socket\") pod \"e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2\" (UID: \"7c9258fe-311f-403d-91f4-5f0708369da4\") " pod="openshift-ingress/e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2" Jul 13 22:24:23.307077 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:23.307051 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c9258fe-311f-403d-91f4-5f0708369da4-workload-socket\") pod \"e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2\" (UID: \"7c9258fe-311f-403d-91f4-5f0708369da4\") " pod="openshift-ingress/e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2" Jul 13 22:24:23.307400 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:23.307378 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c9258fe-311f-403d-91f4-5f0708369da4-istiod-ca-cert\") pod \"e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2\" (UID: \"7c9258fe-311f-403d-91f4-5f0708369da4\") " pod="openshift-ingress/e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2" Jul 13 22:24:23.308881 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:23.308859 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c9258fe-311f-403d-91f4-5f0708369da4-istio-envoy\") pod \"e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2\" (UID: \"7c9258fe-311f-403d-91f4-5f0708369da4\") " pod="openshift-ingress/e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2" Jul 13 22:24:23.309083 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:23.309061 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c9258fe-311f-403d-91f4-5f0708369da4-istio-podinfo\") pod \"e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2\" (UID: \"7c9258fe-311f-403d-91f4-5f0708369da4\") " pod="openshift-ingress/e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2" Jul 13 22:24:23.316808 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:23.316776 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c9258fe-311f-403d-91f4-5f0708369da4-istio-token\") pod \"e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2\" (UID: \"7c9258fe-311f-403d-91f4-5f0708369da4\") " pod="openshift-ingress/e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2" Jul 13 22:24:23.319571 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:23.319542 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qhzr\" (UniqueName: \"kubernetes.io/projected/7c9258fe-311f-403d-91f4-5f0708369da4-kube-api-access-7qhzr\") pod \"e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2\" (UID: \"7c9258fe-311f-403d-91f4-5f0708369da4\") " pod="openshift-ingress/e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2" Jul 13 22:24:23.411010 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:23.410974 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2" Jul 13 22:24:23.560106 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:23.560035 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2"] Jul 13 22:24:23.562421 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:24:23.562399 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c9258fe_311f_403d_91f4_5f0708369da4.slice/crio-232b2bc17892eb8fb769002f1e8a46a752c8cf1148b3759506ed18594165a58f WatchSource:0}: Error finding container 232b2bc17892eb8fb769002f1e8a46a752c8cf1148b3759506ed18594165a58f: Status 404 returned error can't find the container with id 232b2bc17892eb8fb769002f1e8a46a752c8cf1148b3759506ed18594165a58f Jul 13 22:24:23.564420 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:23.564387 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:24:23.564528 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:23.564463 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:24:23.564528 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:23.564507 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:24:24.034250 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:24.034214 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2" event={"ID":"7c9258fe-311f-403d-91f4-5f0708369da4","Type":"ContainerStarted","Data":"22d3468c623822b96917e5c168a162a57b64f44a11af4bb013141a12a99bab5e"} Jul 13 22:24:24.034454 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:24.034264 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2" event={"ID":"7c9258fe-311f-403d-91f4-5f0708369da4","Type":"ContainerStarted","Data":"232b2bc17892eb8fb769002f1e8a46a752c8cf1148b3759506ed18594165a58f"} Jul 13 22:24:24.057481 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:24.057436 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2" podStartSLOduration=1.057420404 podStartE2EDuration="1.057420404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:24:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:24:24.055709343 +0000 UTC m=+1470.049463659" watchObservedRunningTime="2026-07-13 22:24:24.057420404 +0000 UTC m=+1470.051174761" Jul 13 22:24:24.411699 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:24.411663 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2" Jul 13 22:24:24.416590 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:24.416564 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2" Jul 13 22:24:25.039612 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:25.039569 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2" Jul 13 22:24:25.040653 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:25.040636 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2" Jul 13 22:24:29.241337 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:29.241291 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7fae36e-9mb9m"] Jul 13 22:24:29.245403 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:29.245379 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7fae36e-9mb9m" Jul 13 22:24:29.247949 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:29.247928 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 13 22:24:29.248135 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:29.248117 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k5z5d\"" Jul 13 22:24:29.253752 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:29.253397 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7fae36e-9mb9m"] Jul 13 22:24:29.256459 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:29.256435 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97ada3df-ea2a-43bb-9494-8b5cc66e87a1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7fae36e-9mb9m\" (UID: \"97ada3df-ea2a-43bb-9494-8b5cc66e87a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7fae36e-9mb9m" Jul 13 22:24:29.357275 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:29.357195 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97ada3df-ea2a-43bb-9494-8b5cc66e87a1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7fae36e-9mb9m\" (UID: \"97ada3df-ea2a-43bb-9494-8b5cc66e87a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7fae36e-9mb9m" Jul 13 22:24:29.357826 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:29.357806 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97ada3df-ea2a-43bb-9494-8b5cc66e87a1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7fae36e-9mb9m\" (UID: \"97ada3df-ea2a-43bb-9494-8b5cc66e87a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7fae36e-9mb9m" Jul 13 22:24:29.557356 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:29.557260 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7fae36e-9mb9m" Jul 13 22:24:29.754650 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:29.754619 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7fae36e-9mb9m"] Jul 13 22:24:29.756307 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:24:29.756273 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97ada3df_ea2a_43bb_9494_8b5cc66e87a1.slice/crio-ca1e3fa54cf19dd9534ab787f0b45287ba3c5b72b41bd70a19014ba10b71172b WatchSource:0}: Error finding container ca1e3fa54cf19dd9534ab787f0b45287ba3c5b72b41bd70a19014ba10b71172b: Status 404 returned error can't find the container with id ca1e3fa54cf19dd9534ab787f0b45287ba3c5b72b41bd70a19014ba10b71172b Jul 13 22:24:30.068253 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:30.068128 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7fae36e-9mb9m" event={"ID":"97ada3df-ea2a-43bb-9494-8b5cc66e87a1","Type":"ContainerStarted","Data":"ca1e3fa54cf19dd9534ab787f0b45287ba3c5b72b41bd70a19014ba10b71172b"} Jul 13 22:24:31.074314 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:31.074187 2555 generic.go:356] "Generic (PLEG): container finished" podID="97ada3df-ea2a-43bb-9494-8b5cc66e87a1" containerID="31fc2ff7fad80a447bb7ac1e590b4c0a0507a35bee59e1fa74967d8fb6d74f52" exitCode=7 Jul 13 22:24:31.074314 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:31.074272 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7fae36e-9mb9m" event={"ID":"97ada3df-ea2a-43bb-9494-8b5cc66e87a1","Type":"ContainerDied","Data":"31fc2ff7fad80a447bb7ac1e590b4c0a0507a35bee59e1fa74967d8fb6d74f52"} Jul 13 22:24:31.074782 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:31.074507 2555 scope.go:117] "RemoveContainer" containerID="31fc2ff7fad80a447bb7ac1e590b4c0a0507a35bee59e1fa74967d8fb6d74f52" Jul 13 22:24:32.082614 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:32.082579 2555 generic.go:356] "Generic (PLEG): container finished" podID="97ada3df-ea2a-43bb-9494-8b5cc66e87a1" containerID="d76a895961ec04a0977e6deb486432873fb97004505a6e21746342c819dd6dff" exitCode=7 Jul 13 22:24:32.082953 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:32.082618 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7fae36e-9mb9m" event={"ID":"97ada3df-ea2a-43bb-9494-8b5cc66e87a1","Type":"ContainerDied","Data":"d76a895961ec04a0977e6deb486432873fb97004505a6e21746342c819dd6dff"} Jul 13 22:24:32.082953 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:32.082693 2555 scope.go:117] "RemoveContainer" containerID="31fc2ff7fad80a447bb7ac1e590b4c0a0507a35bee59e1fa74967d8fb6d74f52" Jul 13 22:24:32.082953 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:32.082898 2555 scope.go:117] "RemoveContainer" containerID="d76a895961ec04a0977e6deb486432873fb97004505a6e21746342c819dd6dff" Jul 13 22:24:32.083136 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:24:32.083111 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7fae36e-9mb9m_odh-ai-gateway-infra(97ada3df-ea2a-43bb-9494-8b5cc66e87a1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7fae36e-9mb9m" podUID="97ada3df-ea2a-43bb-9494-8b5cc66e87a1" Jul 13 22:24:33.091498 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:33.091470 2555 scope.go:117] "RemoveContainer" containerID="d76a895961ec04a0977e6deb486432873fb97004505a6e21746342c819dd6dff" Jul 13 22:24:33.091858 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:24:33.091704 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7fae36e-9mb9m_odh-ai-gateway-infra(97ada3df-ea2a-43bb-9494-8b5cc66e87a1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7fae36e-9mb9m" podUID="97ada3df-ea2a-43bb-9494-8b5cc66e87a1" Jul 13 22:24:36.110464 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:36.110378 2555 generic.go:356] "Generic (PLEG): container finished" podID="7c9258fe-311f-403d-91f4-5f0708369da4" containerID="22d3468c623822b96917e5c168a162a57b64f44a11af4bb013141a12a99bab5e" exitCode=0 Jul 13 22:24:36.110851 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:36.110458 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2" event={"ID":"7c9258fe-311f-403d-91f4-5f0708369da4","Type":"ContainerDied","Data":"22d3468c623822b96917e5c168a162a57b64f44a11af4bb013141a12a99bab5e"} Jul 13 22:24:36.110851 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:36.110835 2555 scope.go:117] "RemoveContainer" containerID="22d3468c623822b96917e5c168a162a57b64f44a11af4bb013141a12a99bab5e" Jul 13 22:24:36.111943 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:36.111914 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:24:36.112022 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:36.112006 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:24:36.112066 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:36.112044 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:24:37.121953 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:37.121908 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2" event={"ID":"7c9258fe-311f-403d-91f4-5f0708369da4","Type":"ContainerStarted","Data":"aa84df149b14623ff7a12d04e08869c53517836b8563ddaecc22d6fdc26e9af9"} Jul 13 22:24:37.413010 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:37.412974 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2" Jul 13 22:24:37.413441 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:37.413407 2555 patch_prober.go:28] interesting pod/e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 13 22:24:37.413700 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:37.413669 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2" podUID="7c9258fe-311f-403d-91f4-5f0708369da4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 13 22:24:38.411933 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:38.411881 2555 patch_prober.go:28] interesting pod/e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 13 22:24:38.412481 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:38.411979 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2" podUID="7c9258fe-311f-403d-91f4-5f0708369da4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 13 22:24:39.411884 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:39.411834 2555 patch_prober.go:28] interesting pod/e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 13 22:24:39.412248 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:39.412194 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2" podUID="7c9258fe-311f-403d-91f4-5f0708369da4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 13 22:24:40.040114 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:40.040078 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2" Jul 13 22:24:40.412428 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:40.412392 2555 patch_prober.go:28] interesting pod/e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 13 22:24:40.413001 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:40.412968 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2" podUID="7c9258fe-311f-403d-91f4-5f0708369da4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 13 22:24:41.415381 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:41.415344 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2" Jul 13 22:24:41.416382 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:41.416362 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2" Jul 13 22:24:44.556336 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:44.556302 2555 scope.go:117] "RemoveContainer" containerID="d76a895961ec04a0977e6deb486432873fb97004505a6e21746342c819dd6dff" Jul 13 22:24:45.169609 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:45.169573 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7fae36e-9mb9m" event={"ID":"97ada3df-ea2a-43bb-9494-8b5cc66e87a1","Type":"ContainerStarted","Data":"97ff9cc82b146733cc928a5e3269b9b97d6e95cbb29e3ac360f1d4b8c6b82e6b"} Jul 13 22:24:45.188166 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:45.188118 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7fae36e-9mb9m" podStartSLOduration=16.188103302 podStartE2EDuration="16.188103302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:24:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:24:45.185618133 +0000 UTC m=+1491.179372427" watchObservedRunningTime="2026-07-13 22:24:45.188103302 +0000 UTC m=+1491.181857595" Jul 13 22:24:46.176397 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:46.176364 2555 generic.go:356] "Generic (PLEG): container finished" podID="97ada3df-ea2a-43bb-9494-8b5cc66e87a1" containerID="97ff9cc82b146733cc928a5e3269b9b97d6e95cbb29e3ac360f1d4b8c6b82e6b" exitCode=0 Jul 13 22:24:46.176815 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:46.176433 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7fae36e-9mb9m" event={"ID":"97ada3df-ea2a-43bb-9494-8b5cc66e87a1","Type":"ContainerDied","Data":"97ff9cc82b146733cc928a5e3269b9b97d6e95cbb29e3ac360f1d4b8c6b82e6b"} Jul 13 22:24:46.176815 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:46.176486 2555 scope.go:117] "RemoveContainer" containerID="d76a895961ec04a0977e6deb486432873fb97004505a6e21746342c819dd6dff" Jul 13 22:24:46.209571 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:46.209541 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7fae36e-9mb9m"] Jul 13 22:24:47.322989 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:47.322962 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7fae36e-9mb9m" Jul 13 22:24:47.424446 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:47.424415 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97ada3df-ea2a-43bb-9494-8b5cc66e87a1-maas-api-service-ca\") pod \"97ada3df-ea2a-43bb-9494-8b5cc66e87a1\" (UID: \"97ada3df-ea2a-43bb-9494-8b5cc66e87a1\") " Jul 13 22:24:47.424761 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:47.424740 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97ada3df-ea2a-43bb-9494-8b5cc66e87a1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "97ada3df-ea2a-43bb-9494-8b5cc66e87a1" (UID: "97ada3df-ea2a-43bb-9494-8b5cc66e87a1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:24:47.525950 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:47.525871 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97ada3df-ea2a-43bb-9494-8b5cc66e87a1-maas-api-service-ca\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:24:48.189800 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:48.189768 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7fae36e-9mb9m" Jul 13 22:24:48.189800 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:48.189788 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7fae36e-9mb9m" event={"ID":"97ada3df-ea2a-43bb-9494-8b5cc66e87a1","Type":"ContainerDied","Data":"ca1e3fa54cf19dd9534ab787f0b45287ba3c5b72b41bd70a19014ba10b71172b"} Jul 13 22:24:48.189997 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:48.189836 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca1e3fa54cf19dd9534ab787f0b45287ba3c5b72b41bd70a19014ba10b71172b" Jul 13 22:24:48.224298 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:48.224266 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7fae36e-9mb9m"] Jul 13 22:24:48.229784 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:48.229757 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7fae36e-9mb9m"] Jul 13 22:24:48.559367 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:24:48.559286 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97ada3df-ea2a-43bb-9494-8b5cc66e87a1" path="/var/lib/kubelet/pods/97ada3df-ea2a-43bb-9494-8b5cc66e87a1/volumes" Jul 13 22:25:19.291248 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:19.291148 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7fae36e-4lmw9"] Jul 13 22:25:19.291689 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:19.291584 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97ada3df-ea2a-43bb-9494-8b5cc66e87a1" containerName="revoke-keys" Jul 13 22:25:19.291689 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:19.291598 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="97ada3df-ea2a-43bb-9494-8b5cc66e87a1" containerName="revoke-keys" Jul 13 22:25:19.291689 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:19.291605 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97ada3df-ea2a-43bb-9494-8b5cc66e87a1" containerName="revoke-keys" Jul 13 22:25:19.291689 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:19.291612 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="97ada3df-ea2a-43bb-9494-8b5cc66e87a1" containerName="revoke-keys" Jul 13 22:25:19.291689 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:19.291684 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="97ada3df-ea2a-43bb-9494-8b5cc66e87a1" containerName="revoke-keys" Jul 13 22:25:19.291848 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:19.291700 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="97ada3df-ea2a-43bb-9494-8b5cc66e87a1" containerName="revoke-keys" Jul 13 22:25:19.294981 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:19.294963 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7fae36e-4lmw9" Jul 13 22:25:19.297675 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:19.297651 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k5z5d\"" Jul 13 22:25:19.297785 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:19.297662 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 13 22:25:19.303619 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:19.303592 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e65cf22-17d8-4fb6-bbdf-67cd4fd4d296-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7fae36e-4lmw9\" (UID: \"0e65cf22-17d8-4fb6-bbdf-67cd4fd4d296\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7fae36e-4lmw9" Jul 13 22:25:19.303619 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:19.303603 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7fae36e-4lmw9"] Jul 13 22:25:19.404426 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:19.404381 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e65cf22-17d8-4fb6-bbdf-67cd4fd4d296-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7fae36e-4lmw9\" (UID: \"0e65cf22-17d8-4fb6-bbdf-67cd4fd4d296\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7fae36e-4lmw9" Jul 13 22:25:19.405023 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:19.405000 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e65cf22-17d8-4fb6-bbdf-67cd4fd4d296-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7fae36e-4lmw9\" (UID: \"0e65cf22-17d8-4fb6-bbdf-67cd4fd4d296\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7fae36e-4lmw9" Jul 13 22:25:19.605976 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:19.605874 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7fae36e-4lmw9" Jul 13 22:25:19.743917 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:19.743885 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7fae36e-4lmw9"] Jul 13 22:25:19.746073 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:25:19.746043 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e65cf22_17d8_4fb6_bbdf_67cd4fd4d296.slice/crio-a1de935e81ab271aff2d59442a33fdb05576f333d53affe3d3d872173490108b WatchSource:0}: Error finding container a1de935e81ab271aff2d59442a33fdb05576f333d53affe3d3d872173490108b: Status 404 returned error can't find the container with id a1de935e81ab271aff2d59442a33fdb05576f333d53affe3d3d872173490108b Jul 13 22:25:20.361432 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:20.361325 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7fae36e-4lmw9" event={"ID":"0e65cf22-17d8-4fb6-bbdf-67cd4fd4d296","Type":"ContainerStarted","Data":"99ff913d8e63d33abc6cbd25c4a826f543033fae4f8f5f80641c41bafd30eeba"} Jul 13 22:25:20.361432 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:20.361414 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7fae36e-4lmw9" event={"ID":"0e65cf22-17d8-4fb6-bbdf-67cd4fd4d296","Type":"ContainerStarted","Data":"a1de935e81ab271aff2d59442a33fdb05576f333d53affe3d3d872173490108b"} Jul 13 22:25:20.378711 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:20.378663 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7fae36e-4lmw9" podStartSLOduration=1.37864801 podStartE2EDuration="1.37864801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:25:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:25:20.376741059 +0000 UTC m=+1526.370495364" watchObservedRunningTime="2026-07-13 22:25:20.37864801 +0000 UTC m=+1526.372402354" Jul 13 22:25:21.368715 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:21.368676 2555 generic.go:356] "Generic (PLEG): container finished" podID="0e65cf22-17d8-4fb6-bbdf-67cd4fd4d296" containerID="99ff913d8e63d33abc6cbd25c4a826f543033fae4f8f5f80641c41bafd30eeba" exitCode=0 Jul 13 22:25:21.369101 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:21.368737 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7fae36e-4lmw9" event={"ID":"0e65cf22-17d8-4fb6-bbdf-67cd4fd4d296","Type":"ContainerDied","Data":"99ff913d8e63d33abc6cbd25c4a826f543033fae4f8f5f80641c41bafd30eeba"} Jul 13 22:25:22.515016 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:22.514992 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7fae36e-4lmw9" Jul 13 22:25:22.532320 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:22.532286 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e65cf22-17d8-4fb6-bbdf-67cd4fd4d296-maas-api-service-ca\") pod \"0e65cf22-17d8-4fb6-bbdf-67cd4fd4d296\" (UID: \"0e65cf22-17d8-4fb6-bbdf-67cd4fd4d296\") " Jul 13 22:25:22.532710 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:22.532665 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e65cf22-17d8-4fb6-bbdf-67cd4fd4d296-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0e65cf22-17d8-4fb6-bbdf-67cd4fd4d296" (UID: "0e65cf22-17d8-4fb6-bbdf-67cd4fd4d296"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:25:22.633806 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:22.633720 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e65cf22-17d8-4fb6-bbdf-67cd4fd4d296-maas-api-service-ca\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:25:23.380646 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:23.380610 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7fae36e-4lmw9" event={"ID":"0e65cf22-17d8-4fb6-bbdf-67cd4fd4d296","Type":"ContainerDied","Data":"a1de935e81ab271aff2d59442a33fdb05576f333d53affe3d3d872173490108b"} Jul 13 22:25:23.380646 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:23.380629 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7fae36e-4lmw9" Jul 13 22:25:23.380852 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:23.380662 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1de935e81ab271aff2d59442a33fdb05576f333d53affe3d3d872173490108b" Jul 13 22:25:44.630013 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:44.629973 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2"] Jul 13 22:25:44.630435 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:44.630192 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2" podUID="7c9258fe-311f-403d-91f4-5f0708369da4" containerName="istio-proxy" containerID="cri-o://aa84df149b14623ff7a12d04e08869c53517836b8563ddaecc22d6fdc26e9af9" gracePeriod=30 Jul 13 22:25:45.195876 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:45.195844 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b"] Jul 13 22:25:45.196316 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:45.196298 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e65cf22-17d8-4fb6-bbdf-67cd4fd4d296" containerName="revoke-keys" Jul 13 22:25:45.196316 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:45.196318 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e65cf22-17d8-4fb6-bbdf-67cd4fd4d296" containerName="revoke-keys" Jul 13 22:25:45.196466 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:45.196331 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97ada3df-ea2a-43bb-9494-8b5cc66e87a1" containerName="revoke-keys" Jul 13 22:25:45.196466 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:45.196339 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="97ada3df-ea2a-43bb-9494-8b5cc66e87a1" containerName="revoke-keys" Jul 13 22:25:45.196466 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:45.196458 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="97ada3df-ea2a-43bb-9494-8b5cc66e87a1" containerName="revoke-keys" Jul 13 22:25:45.196614 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:45.196477 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e65cf22-17d8-4fb6-bbdf-67cd4fd4d296" containerName="revoke-keys" Jul 13 22:25:45.199602 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:45.199581 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b" Jul 13 22:25:45.202867 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:45.202842 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-65448f9f-gw-openshift-default-dockercfg-gvzwx\"" Jul 13 22:25:45.219099 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:45.219066 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b"] Jul 13 22:25:45.339428 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:45.339389 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93867ce3-9701-406e-94cf-6c30c413a3dc-istio-token\") pod \"e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b\" (UID: \"93867ce3-9701-406e-94cf-6c30c413a3dc\") " pod="openshift-ingress/e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b" Jul 13 22:25:45.339643 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:45.339453 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93867ce3-9701-406e-94cf-6c30c413a3dc-istio-podinfo\") pod \"e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b\" (UID: \"93867ce3-9701-406e-94cf-6c30c413a3dc\") " pod="openshift-ingress/e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b" Jul 13 22:25:45.339643 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:45.339502 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kmrk\" (UniqueName: \"kubernetes.io/projected/93867ce3-9701-406e-94cf-6c30c413a3dc-kube-api-access-7kmrk\") pod \"e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b\" (UID: \"93867ce3-9701-406e-94cf-6c30c413a3dc\") " pod="openshift-ingress/e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b" Jul 13 22:25:45.339643 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:45.339554 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93867ce3-9701-406e-94cf-6c30c413a3dc-istio-envoy\") pod \"e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b\" (UID: \"93867ce3-9701-406e-94cf-6c30c413a3dc\") " pod="openshift-ingress/e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b" Jul 13 22:25:45.339643 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:45.339601 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93867ce3-9701-406e-94cf-6c30c413a3dc-workload-certs\") pod \"e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b\" (UID: \"93867ce3-9701-406e-94cf-6c30c413a3dc\") " pod="openshift-ingress/e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b" Jul 13 22:25:45.339643 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:45.339640 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93867ce3-9701-406e-94cf-6c30c413a3dc-credential-socket\") pod \"e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b\" (UID: \"93867ce3-9701-406e-94cf-6c30c413a3dc\") " pod="openshift-ingress/e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b" Jul 13 22:25:45.339917 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:45.339681 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93867ce3-9701-406e-94cf-6c30c413a3dc-istiod-ca-cert\") pod \"e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b\" (UID: \"93867ce3-9701-406e-94cf-6c30c413a3dc\") " pod="openshift-ingress/e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b" Jul 13 22:25:45.339917 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:45.339719 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93867ce3-9701-406e-94cf-6c30c413a3dc-istio-data\") pod \"e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b\" (UID: \"93867ce3-9701-406e-94cf-6c30c413a3dc\") " pod="openshift-ingress/e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b" Jul 13 22:25:45.339917 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:45.339741 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93867ce3-9701-406e-94cf-6c30c413a3dc-workload-socket\") pod \"e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b\" (UID: \"93867ce3-9701-406e-94cf-6c30c413a3dc\") " pod="openshift-ingress/e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b" Jul 13 22:25:45.440602 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:45.440567 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93867ce3-9701-406e-94cf-6c30c413a3dc-istiod-ca-cert\") pod \"e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b\" (UID: \"93867ce3-9701-406e-94cf-6c30c413a3dc\") " pod="openshift-ingress/e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b" Jul 13 22:25:45.440602 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:45.440606 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93867ce3-9701-406e-94cf-6c30c413a3dc-istio-data\") pod \"e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b\" (UID: \"93867ce3-9701-406e-94cf-6c30c413a3dc\") " pod="openshift-ingress/e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b" Jul 13 22:25:45.441007 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:45.440632 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93867ce3-9701-406e-94cf-6c30c413a3dc-workload-socket\") pod \"e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b\" (UID: \"93867ce3-9701-406e-94cf-6c30c413a3dc\") " pod="openshift-ingress/e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b" Jul 13 22:25:45.441007 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:45.440683 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93867ce3-9701-406e-94cf-6c30c413a3dc-istio-token\") pod \"e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b\" (UID: \"93867ce3-9701-406e-94cf-6c30c413a3dc\") " pod="openshift-ingress/e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b" Jul 13 22:25:45.441007 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:45.440714 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93867ce3-9701-406e-94cf-6c30c413a3dc-istio-podinfo\") pod \"e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b\" (UID: \"93867ce3-9701-406e-94cf-6c30c413a3dc\") " pod="openshift-ingress/e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b" Jul 13 22:25:45.441007 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:45.440751 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kmrk\" (UniqueName: \"kubernetes.io/projected/93867ce3-9701-406e-94cf-6c30c413a3dc-kube-api-access-7kmrk\") pod \"e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b\" (UID: \"93867ce3-9701-406e-94cf-6c30c413a3dc\") " pod="openshift-ingress/e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b" Jul 13 22:25:45.441007 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:45.440792 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93867ce3-9701-406e-94cf-6c30c413a3dc-istio-envoy\") pod \"e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b\" (UID: \"93867ce3-9701-406e-94cf-6c30c413a3dc\") " pod="openshift-ingress/e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b" Jul 13 22:25:45.441007 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:45.440829 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93867ce3-9701-406e-94cf-6c30c413a3dc-workload-certs\") pod \"e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b\" (UID: \"93867ce3-9701-406e-94cf-6c30c413a3dc\") " pod="openshift-ingress/e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b" Jul 13 22:25:45.441007 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:45.440869 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93867ce3-9701-406e-94cf-6c30c413a3dc-credential-socket\") pod \"e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b\" (UID: \"93867ce3-9701-406e-94cf-6c30c413a3dc\") " pod="openshift-ingress/e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b" Jul 13 22:25:45.441343 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:45.441084 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93867ce3-9701-406e-94cf-6c30c413a3dc-istio-data\") pod \"e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b\" (UID: \"93867ce3-9701-406e-94cf-6c30c413a3dc\") " pod="openshift-ingress/e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b" Jul 13 22:25:45.441343 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:45.441121 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93867ce3-9701-406e-94cf-6c30c413a3dc-workload-socket\") pod \"e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b\" (UID: \"93867ce3-9701-406e-94cf-6c30c413a3dc\") " pod="openshift-ingress/e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b" Jul 13 22:25:45.441343 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:45.441270 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93867ce3-9701-406e-94cf-6c30c413a3dc-credential-socket\") pod \"e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b\" (UID: \"93867ce3-9701-406e-94cf-6c30c413a3dc\") " pod="openshift-ingress/e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b" Jul 13 22:25:45.441444 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:45.441369 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93867ce3-9701-406e-94cf-6c30c413a3dc-istiod-ca-cert\") pod \"e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b\" (UID: \"93867ce3-9701-406e-94cf-6c30c413a3dc\") " pod="openshift-ingress/e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b" Jul 13 22:25:45.441444 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:45.441398 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93867ce3-9701-406e-94cf-6c30c413a3dc-workload-certs\") pod \"e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b\" (UID: \"93867ce3-9701-406e-94cf-6c30c413a3dc\") " pod="openshift-ingress/e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b" Jul 13 22:25:45.443277 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:45.443249 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93867ce3-9701-406e-94cf-6c30c413a3dc-istio-envoy\") pod \"e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b\" (UID: \"93867ce3-9701-406e-94cf-6c30c413a3dc\") " pod="openshift-ingress/e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b" Jul 13 22:25:45.443368 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:45.443346 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93867ce3-9701-406e-94cf-6c30c413a3dc-istio-podinfo\") pod \"e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b\" (UID: \"93867ce3-9701-406e-94cf-6c30c413a3dc\") " pod="openshift-ingress/e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b" Jul 13 22:25:45.450911 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:45.450835 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kmrk\" (UniqueName: \"kubernetes.io/projected/93867ce3-9701-406e-94cf-6c30c413a3dc-kube-api-access-7kmrk\") pod \"e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b\" (UID: \"93867ce3-9701-406e-94cf-6c30c413a3dc\") " pod="openshift-ingress/e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b" Jul 13 22:25:45.451511 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:45.451489 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93867ce3-9701-406e-94cf-6c30c413a3dc-istio-token\") pod \"e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b\" (UID: \"93867ce3-9701-406e-94cf-6c30c413a3dc\") " pod="openshift-ingress/e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b" Jul 13 22:25:45.510667 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:45.510623 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b" Jul 13 22:25:45.659443 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:45.659414 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b"] Jul 13 22:25:45.661882 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:25:45.661857 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93867ce3_9701_406e_94cf_6c30c413a3dc.slice/crio-59620a54368757b2a1121301ef9d3c43f084e69df425cf8781452a7e0bee1eff WatchSource:0}: Error finding container 59620a54368757b2a1121301ef9d3c43f084e69df425cf8781452a7e0bee1eff: Status 404 returned error can't find the container with id 59620a54368757b2a1121301ef9d3c43f084e69df425cf8781452a7e0bee1eff Jul 13 22:25:45.663795 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:45.663761 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:25:45.663883 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:45.663829 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:25:45.663883 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:45.663859 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:25:46.507630 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:46.507595 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b" event={"ID":"93867ce3-9701-406e-94cf-6c30c413a3dc","Type":"ContainerStarted","Data":"ef641058c01c317cfea88288b3c388b5d10a4d8d49897506fdc7b28fd62e9643"} Jul 13 22:25:46.507813 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:46.507655 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b" event={"ID":"93867ce3-9701-406e-94cf-6c30c413a3dc","Type":"ContainerStarted","Data":"59620a54368757b2a1121301ef9d3c43f084e69df425cf8781452a7e0bee1eff"} Jul 13 22:25:46.511697 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:46.511672 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b" Jul 13 22:25:46.513169 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:46.513148 2555 patch_prober.go:28] interesting pod/e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 13 22:25:46.513263 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:46.513193 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b" podUID="93867ce3-9701-406e-94cf-6c30c413a3dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 13 22:25:46.536174 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:46.536122 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b" podStartSLOduration=1.536104585 podStartE2EDuration="1.536104585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:25:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:25:46.534668658 +0000 UTC m=+1552.528422975" watchObservedRunningTime="2026-07-13 22:25:46.536104585 +0000 UTC m=+1552.529858877" Jul 13 22:25:47.515088 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:47.515056 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b" Jul 13 22:25:48.519263 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:48.519229 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b" Jul 13 22:25:48.520042 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:48.520023 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b" Jul 13 22:25:49.883487 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:49.883460 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2" Jul 13 22:25:49.986593 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:49.986559 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c9258fe-311f-403d-91f4-5f0708369da4-istiod-ca-cert\") pod \"7c9258fe-311f-403d-91f4-5f0708369da4\" (UID: \"7c9258fe-311f-403d-91f4-5f0708369da4\") " Jul 13 22:25:49.986757 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:49.986648 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7qhzr\" (UniqueName: \"kubernetes.io/projected/7c9258fe-311f-403d-91f4-5f0708369da4-kube-api-access-7qhzr\") pod \"7c9258fe-311f-403d-91f4-5f0708369da4\" (UID: \"7c9258fe-311f-403d-91f4-5f0708369da4\") " Jul 13 22:25:49.986757 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:49.986680 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c9258fe-311f-403d-91f4-5f0708369da4-istio-token\") pod \"7c9258fe-311f-403d-91f4-5f0708369da4\" (UID: \"7c9258fe-311f-403d-91f4-5f0708369da4\") " Jul 13 22:25:49.986757 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:49.986712 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c9258fe-311f-403d-91f4-5f0708369da4-workload-socket\") pod \"7c9258fe-311f-403d-91f4-5f0708369da4\" (UID: \"7c9258fe-311f-403d-91f4-5f0708369da4\") " Jul 13 22:25:49.986937 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:49.986757 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c9258fe-311f-403d-91f4-5f0708369da4-istio-envoy\") pod \"7c9258fe-311f-403d-91f4-5f0708369da4\" (UID: \"7c9258fe-311f-403d-91f4-5f0708369da4\") " Jul 13 22:25:49.986937 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:49.986803 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c9258fe-311f-403d-91f4-5f0708369da4-istio-podinfo\") pod \"7c9258fe-311f-403d-91f4-5f0708369da4\" (UID: \"7c9258fe-311f-403d-91f4-5f0708369da4\") " Jul 13 22:25:49.986937 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:49.986830 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c9258fe-311f-403d-91f4-5f0708369da4-credential-socket\") pod \"7c9258fe-311f-403d-91f4-5f0708369da4\" (UID: \"7c9258fe-311f-403d-91f4-5f0708369da4\") " Jul 13 22:25:49.986937 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:49.986876 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c9258fe-311f-403d-91f4-5f0708369da4-workload-certs\") pod \"7c9258fe-311f-403d-91f4-5f0708369da4\" (UID: \"7c9258fe-311f-403d-91f4-5f0708369da4\") " Jul 13 22:25:49.986937 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:49.986905 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c9258fe-311f-403d-91f4-5f0708369da4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7c9258fe-311f-403d-91f4-5f0708369da4" (UID: "7c9258fe-311f-403d-91f4-5f0708369da4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:25:49.986937 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:49.986920 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c9258fe-311f-403d-91f4-5f0708369da4-istio-data\") pod \"7c9258fe-311f-403d-91f4-5f0708369da4\" (UID: \"7c9258fe-311f-403d-91f4-5f0708369da4\") " Jul 13 22:25:49.987296 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:49.987187 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c9258fe-311f-403d-91f4-5f0708369da4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7c9258fe-311f-403d-91f4-5f0708369da4" (UID: "7c9258fe-311f-403d-91f4-5f0708369da4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:25:49.987510 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:49.987404 2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c9258fe-311f-403d-91f4-5f0708369da4-istiod-ca-cert\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:25:49.987510 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:49.987467 2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c9258fe-311f-403d-91f4-5f0708369da4-credential-socket\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:25:49.987923 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:49.987471 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c9258fe-311f-403d-91f4-5f0708369da4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7c9258fe-311f-403d-91f4-5f0708369da4" (UID: "7c9258fe-311f-403d-91f4-5f0708369da4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:25:49.987923 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:49.987811 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c9258fe-311f-403d-91f4-5f0708369da4-istio-data" (OuterVolumeSpecName: "istio-data") pod "7c9258fe-311f-403d-91f4-5f0708369da4" (UID: "7c9258fe-311f-403d-91f4-5f0708369da4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:25:49.988058 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:49.987953 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c9258fe-311f-403d-91f4-5f0708369da4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7c9258fe-311f-403d-91f4-5f0708369da4" (UID: "7c9258fe-311f-403d-91f4-5f0708369da4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:25:49.989169 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:49.989133 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c9258fe-311f-403d-91f4-5f0708369da4-istio-token" (OuterVolumeSpecName: "istio-token") pod "7c9258fe-311f-403d-91f4-5f0708369da4" (UID: "7c9258fe-311f-403d-91f4-5f0708369da4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:25:49.989627 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:49.989599 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c9258fe-311f-403d-91f4-5f0708369da4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7c9258fe-311f-403d-91f4-5f0708369da4" (UID: "7c9258fe-311f-403d-91f4-5f0708369da4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:25:49.989945 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:49.989919 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c9258fe-311f-403d-91f4-5f0708369da4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7c9258fe-311f-403d-91f4-5f0708369da4" (UID: "7c9258fe-311f-403d-91f4-5f0708369da4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 13 22:25:49.990027 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:49.989925 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c9258fe-311f-403d-91f4-5f0708369da4-kube-api-access-7qhzr" (OuterVolumeSpecName: "kube-api-access-7qhzr") pod "7c9258fe-311f-403d-91f4-5f0708369da4" (UID: "7c9258fe-311f-403d-91f4-5f0708369da4"). InnerVolumeSpecName "kube-api-access-7qhzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:25:50.088101 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:50.088064 2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c9258fe-311f-403d-91f4-5f0708369da4-workload-certs\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:25:50.088101 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:50.088096 2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c9258fe-311f-403d-91f4-5f0708369da4-istio-data\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:25:50.088101 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:50.088104 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qhzr\" (UniqueName: \"kubernetes.io/projected/7c9258fe-311f-403d-91f4-5f0708369da4-kube-api-access-7qhzr\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:25:50.088439 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:50.088113 2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c9258fe-311f-403d-91f4-5f0708369da4-istio-token\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:25:50.088439 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:50.088123 2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c9258fe-311f-403d-91f4-5f0708369da4-workload-socket\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:25:50.088439 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:50.088132 2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c9258fe-311f-403d-91f4-5f0708369da4-istio-envoy\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:25:50.088439 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:50.088140 2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c9258fe-311f-403d-91f4-5f0708369da4-istio-podinfo\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:25:50.530975 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:50.530941 2555 generic.go:356] "Generic (PLEG): container finished" podID="7c9258fe-311f-403d-91f4-5f0708369da4" containerID="aa84df149b14623ff7a12d04e08869c53517836b8563ddaecc22d6fdc26e9af9" exitCode=0 Jul 13 22:25:50.531166 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:50.530988 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2" Jul 13 22:25:50.531166 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:50.531025 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2" event={"ID":"7c9258fe-311f-403d-91f4-5f0708369da4","Type":"ContainerDied","Data":"aa84df149b14623ff7a12d04e08869c53517836b8563ddaecc22d6fdc26e9af9"} Jul 13 22:25:50.531166 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:50.531082 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2" event={"ID":"7c9258fe-311f-403d-91f4-5f0708369da4","Type":"ContainerDied","Data":"232b2bc17892eb8fb769002f1e8a46a752c8cf1148b3759506ed18594165a58f"} Jul 13 22:25:50.531166 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:50.531108 2555 scope.go:117] "RemoveContainer" containerID="aa84df149b14623ff7a12d04e08869c53517836b8563ddaecc22d6fdc26e9af9" Jul 13 22:25:50.541355 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:50.541336 2555 scope.go:117] "RemoveContainer" containerID="22d3468c623822b96917e5c168a162a57b64f44a11af4bb013141a12a99bab5e" Jul 13 22:25:50.549733 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:50.549717 2555 scope.go:117] "RemoveContainer" containerID="aa84df149b14623ff7a12d04e08869c53517836b8563ddaecc22d6fdc26e9af9" Jul 13 22:25:50.550008 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:25:50.549989 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa84df149b14623ff7a12d04e08869c53517836b8563ddaecc22d6fdc26e9af9\": container with ID starting with aa84df149b14623ff7a12d04e08869c53517836b8563ddaecc22d6fdc26e9af9 not found: ID does not exist" containerID="aa84df149b14623ff7a12d04e08869c53517836b8563ddaecc22d6fdc26e9af9" Jul 13 22:25:50.550047 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:50.550016 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa84df149b14623ff7a12d04e08869c53517836b8563ddaecc22d6fdc26e9af9"} err="failed to get container status \"aa84df149b14623ff7a12d04e08869c53517836b8563ddaecc22d6fdc26e9af9\": rpc error: code = NotFound desc = could not find container \"aa84df149b14623ff7a12d04e08869c53517836b8563ddaecc22d6fdc26e9af9\": container with ID starting with aa84df149b14623ff7a12d04e08869c53517836b8563ddaecc22d6fdc26e9af9 not found: ID does not exist" Jul 13 22:25:50.550047 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:50.550032 2555 scope.go:117] "RemoveContainer" containerID="22d3468c623822b96917e5c168a162a57b64f44a11af4bb013141a12a99bab5e" Jul 13 22:25:50.550294 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:25:50.550276 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22d3468c623822b96917e5c168a162a57b64f44a11af4bb013141a12a99bab5e\": container with ID starting with 22d3468c623822b96917e5c168a162a57b64f44a11af4bb013141a12a99bab5e not found: ID does not exist" containerID="22d3468c623822b96917e5c168a162a57b64f44a11af4bb013141a12a99bab5e" Jul 13 22:25:50.550338 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:50.550301 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22d3468c623822b96917e5c168a162a57b64f44a11af4bb013141a12a99bab5e"} err="failed to get container status \"22d3468c623822b96917e5c168a162a57b64f44a11af4bb013141a12a99bab5e\": rpc error: code = NotFound desc = could not find container \"22d3468c623822b96917e5c168a162a57b64f44a11af4bb013141a12a99bab5e\": container with ID starting with 22d3468c623822b96917e5c168a162a57b64f44a11af4bb013141a12a99bab5e not found: ID does not exist" Jul 13 22:25:50.560822 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:50.560800 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2"] Jul 13 22:25:50.563338 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:50.563321 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-d7fae36e-openshift-default-f944699bf-vhlx2"] Jul 13 22:25:50.947919 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:50.947882 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65448f9f-nlwmf"] Jul 13 22:25:50.948310 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:50.948287 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c9258fe-311f-403d-91f4-5f0708369da4" containerName="istio-proxy" Jul 13 22:25:50.948310 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:50.948302 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c9258fe-311f-403d-91f4-5f0708369da4" containerName="istio-proxy" Jul 13 22:25:50.948387 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:50.948317 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c9258fe-311f-403d-91f4-5f0708369da4" containerName="istio-proxy" Jul 13 22:25:50.948387 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:50.948323 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c9258fe-311f-403d-91f4-5f0708369da4" containerName="istio-proxy" Jul 13 22:25:50.948387 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:50.948377 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c9258fe-311f-403d-91f4-5f0708369da4" containerName="istio-proxy" Jul 13 22:25:50.948387 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:50.948385 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c9258fe-311f-403d-91f4-5f0708369da4" containerName="istio-proxy" Jul 13 22:25:50.952844 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:50.952825 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65448f9f-nlwmf" Jul 13 22:25:50.955486 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:50.955457 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 13 22:25:50.955590 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:50.955570 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k5z5d\"" Jul 13 22:25:50.959528 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:50.959508 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65448f9f-nlwmf"] Jul 13 22:25:51.098059 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:51.098020 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbd8b1f6-e75c-42af-8ba1-1b7694dea93c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-65448f9f-nlwmf\" (UID: \"cbd8b1f6-e75c-42af-8ba1-1b7694dea93c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65448f9f-nlwmf" Jul 13 22:25:51.200353 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:51.199790 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbd8b1f6-e75c-42af-8ba1-1b7694dea93c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-65448f9f-nlwmf\" (UID: \"cbd8b1f6-e75c-42af-8ba1-1b7694dea93c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65448f9f-nlwmf" Jul 13 22:25:51.200676 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:51.200651 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbd8b1f6-e75c-42af-8ba1-1b7694dea93c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-65448f9f-nlwmf\" (UID: \"cbd8b1f6-e75c-42af-8ba1-1b7694dea93c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65448f9f-nlwmf" Jul 13 22:25:51.264409 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:51.264376 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65448f9f-nlwmf" Jul 13 22:25:51.400931 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:51.400703 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65448f9f-nlwmf"] Jul 13 22:25:51.403373 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:25:51.403346 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbd8b1f6_e75c_42af_8ba1_1b7694dea93c.slice/crio-00c9903e1245c24df5a39fe2cb38b7592f29b874231edae1039355c3705666f9 WatchSource:0}: Error finding container 00c9903e1245c24df5a39fe2cb38b7592f29b874231edae1039355c3705666f9: Status 404 returned error can't find the container with id 00c9903e1245c24df5a39fe2cb38b7592f29b874231edae1039355c3705666f9 Jul 13 22:25:51.538114 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:51.538025 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65448f9f-nlwmf" event={"ID":"cbd8b1f6-e75c-42af-8ba1-1b7694dea93c","Type":"ContainerStarted","Data":"00c9903e1245c24df5a39fe2cb38b7592f29b874231edae1039355c3705666f9"} Jul 13 22:25:52.545074 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:52.545043 2555 generic.go:356] "Generic (PLEG): container finished" podID="cbd8b1f6-e75c-42af-8ba1-1b7694dea93c" containerID="6c986c56042f12726ff22941759b612b0253c3d28b9be158704fd0d452069504" exitCode=7 Jul 13 22:25:52.545577 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:52.545136 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65448f9f-nlwmf" event={"ID":"cbd8b1f6-e75c-42af-8ba1-1b7694dea93c","Type":"ContainerDied","Data":"6c986c56042f12726ff22941759b612b0253c3d28b9be158704fd0d452069504"} Jul 13 22:25:52.545577 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:52.545358 2555 scope.go:117] "RemoveContainer" containerID="6c986c56042f12726ff22941759b612b0253c3d28b9be158704fd0d452069504" Jul 13 22:25:52.563914 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:52.563878 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c9258fe-311f-403d-91f4-5f0708369da4" path="/var/lib/kubelet/pods/7c9258fe-311f-403d-91f4-5f0708369da4/volumes" Jul 13 22:25:53.552090 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:53.552052 2555 generic.go:356] "Generic (PLEG): container finished" podID="cbd8b1f6-e75c-42af-8ba1-1b7694dea93c" containerID="290d4733a64805b3714a8be4e468d8b3fe4870d0c35b680deedef812af60ff66" exitCode=7 Jul 13 22:25:53.552532 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:53.552135 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65448f9f-nlwmf" event={"ID":"cbd8b1f6-e75c-42af-8ba1-1b7694dea93c","Type":"ContainerDied","Data":"290d4733a64805b3714a8be4e468d8b3fe4870d0c35b680deedef812af60ff66"} Jul 13 22:25:53.552532 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:53.552195 2555 scope.go:117] "RemoveContainer" containerID="6c986c56042f12726ff22941759b612b0253c3d28b9be158704fd0d452069504" Jul 13 22:25:53.552532 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:53.552352 2555 scope.go:117] "RemoveContainer" containerID="290d4733a64805b3714a8be4e468d8b3fe4870d0c35b680deedef812af60ff66" Jul 13 22:25:53.552674 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:25:53.552580 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-65448f9f-nlwmf_odh-ai-gateway-infra(cbd8b1f6-e75c-42af-8ba1-1b7694dea93c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65448f9f-nlwmf" podUID="cbd8b1f6-e75c-42af-8ba1-1b7694dea93c" Jul 13 22:25:54.560710 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:25:54.560637 2555 scope.go:117] "RemoveContainer" containerID="290d4733a64805b3714a8be4e468d8b3fe4870d0c35b680deedef812af60ff66" Jul 13 22:25:54.561055 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:25:54.560813 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-65448f9f-nlwmf_odh-ai-gateway-infra(cbd8b1f6-e75c-42af-8ba1-1b7694dea93c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65448f9f-nlwmf" podUID="cbd8b1f6-e75c-42af-8ba1-1b7694dea93c" Jul 13 22:26:00.595113 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:00.595076 2555 generic.go:356] "Generic (PLEG): container finished" podID="93867ce3-9701-406e-94cf-6c30c413a3dc" containerID="ef641058c01c317cfea88288b3c388b5d10a4d8d49897506fdc7b28fd62e9643" exitCode=0 Jul 13 22:26:00.595557 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:00.595161 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b" event={"ID":"93867ce3-9701-406e-94cf-6c30c413a3dc","Type":"ContainerDied","Data":"ef641058c01c317cfea88288b3c388b5d10a4d8d49897506fdc7b28fd62e9643"} Jul 13 22:26:00.595616 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:00.595600 2555 scope.go:117] "RemoveContainer" containerID="ef641058c01c317cfea88288b3c388b5d10a4d8d49897506fdc7b28fd62e9643" Jul 13 22:26:00.596659 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:00.596626 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:26:00.596740 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:00.596727 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:26:00.596795 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:00.596783 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:26:01.608187 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:01.607166 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b" event={"ID":"93867ce3-9701-406e-94cf-6c30c413a3dc","Type":"ContainerStarted","Data":"4083bb810a429fa2a34781b28906661bcb77e1b3d1062dddb2b5c18f328fa4bf"} Jul 13 22:26:02.512849 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:02.512816 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b" Jul 13 22:26:02.513316 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:02.513283 2555 patch_prober.go:28] interesting pod/e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 13 22:26:02.513531 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:02.513494 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b" podUID="93867ce3-9701-406e-94cf-6c30c413a3dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 13 22:26:03.511260 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:03.511194 2555 patch_prober.go:28] interesting pod/e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 13 22:26:03.511836 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:03.511804 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b" podUID="93867ce3-9701-406e-94cf-6c30c413a3dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 13 22:26:03.519649 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:03.519608 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b" Jul 13 22:26:04.511788 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:04.511739 2555 patch_prober.go:28] interesting pod/e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 13 22:26:04.512313 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:04.511832 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b" podUID="93867ce3-9701-406e-94cf-6c30c413a3dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 13 22:26:05.515527 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:05.515495 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b" Jul 13 22:26:05.516537 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:05.516516 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b" Jul 13 22:26:07.553617 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:07.553587 2555 scope.go:117] "RemoveContainer" containerID="290d4733a64805b3714a8be4e468d8b3fe4870d0c35b680deedef812af60ff66" Jul 13 22:26:08.652989 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:08.652960 2555 generic.go:356] "Generic (PLEG): container finished" podID="cbd8b1f6-e75c-42af-8ba1-1b7694dea93c" containerID="bbd6c0e5171d02427dffa4c69559629d9b0ea94390f3420a443afae269e85562" exitCode=0 Jul 13 22:26:08.653455 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:08.653013 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65448f9f-nlwmf" event={"ID":"cbd8b1f6-e75c-42af-8ba1-1b7694dea93c","Type":"ContainerDied","Data":"bbd6c0e5171d02427dffa4c69559629d9b0ea94390f3420a443afae269e85562"} Jul 13 22:26:08.653455 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:08.653065 2555 scope.go:117] "RemoveContainer" containerID="290d4733a64805b3714a8be4e468d8b3fe4870d0c35b680deedef812af60ff66" Jul 13 22:26:09.689834 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:09.689795 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65448f9f-nlwmf"] Jul 13 22:26:09.798032 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:09.798011 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65448f9f-nlwmf" Jul 13 22:26:09.867480 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:09.867448 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbd8b1f6-e75c-42af-8ba1-1b7694dea93c-maas-api-service-ca\") pod \"cbd8b1f6-e75c-42af-8ba1-1b7694dea93c\" (UID: \"cbd8b1f6-e75c-42af-8ba1-1b7694dea93c\") " Jul 13 22:26:09.867833 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:09.867808 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbd8b1f6-e75c-42af-8ba1-1b7694dea93c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cbd8b1f6-e75c-42af-8ba1-1b7694dea93c" (UID: "cbd8b1f6-e75c-42af-8ba1-1b7694dea93c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:26:09.968326 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:09.968239 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbd8b1f6-e75c-42af-8ba1-1b7694dea93c-maas-api-service-ca\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:26:10.667610 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:10.667580 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65448f9f-nlwmf" Jul 13 22:26:10.667610 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:10.667591 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65448f9f-nlwmf" event={"ID":"cbd8b1f6-e75c-42af-8ba1-1b7694dea93c","Type":"ContainerDied","Data":"00c9903e1245c24df5a39fe2cb38b7592f29b874231edae1039355c3705666f9"} Jul 13 22:26:10.667849 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:10.667646 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00c9903e1245c24df5a39fe2cb38b7592f29b874231edae1039355c3705666f9" Jul 13 22:26:10.698270 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:10.698235 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65448f9f-nlwmf"] Jul 13 22:26:10.702593 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:10.702572 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65448f9f-nlwmf"] Jul 13 22:26:12.560038 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:12.560002 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbd8b1f6-e75c-42af-8ba1-1b7694dea93c" path="/var/lib/kubelet/pods/cbd8b1f6-e75c-42af-8ba1-1b7694dea93c/volumes" Jul 13 22:26:40.997655 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:40.997616 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65448f9f-7m9w2"] Jul 13 22:26:40.998080 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:40.998000 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbd8b1f6-e75c-42af-8ba1-1b7694dea93c" containerName="revoke-keys" Jul 13 22:26:40.998080 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:40.998011 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbd8b1f6-e75c-42af-8ba1-1b7694dea93c" containerName="revoke-keys" Jul 13 22:26:40.998080 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:40.998032 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbd8b1f6-e75c-42af-8ba1-1b7694dea93c" containerName="revoke-keys" Jul 13 22:26:40.998080 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:40.998037 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbd8b1f6-e75c-42af-8ba1-1b7694dea93c" containerName="revoke-keys" Jul 13 22:26:40.998229 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:40.998096 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbd8b1f6-e75c-42af-8ba1-1b7694dea93c" containerName="revoke-keys" Jul 13 22:26:40.998229 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:40.998103 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbd8b1f6-e75c-42af-8ba1-1b7694dea93c" containerName="revoke-keys" Jul 13 22:26:40.998229 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:40.998111 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbd8b1f6-e75c-42af-8ba1-1b7694dea93c" containerName="revoke-keys" Jul 13 22:26:41.001391 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:41.001373 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65448f9f-7m9w2" Jul 13 22:26:41.004056 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:41.004033 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k5z5d\"" Jul 13 22:26:41.004180 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:41.004058 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 13 22:26:41.010063 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:41.010039 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65448f9f-7m9w2"] Jul 13 22:26:41.148878 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:41.148837 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d47e57de-4b10-40f8-99c4-b127c59c8930-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-65448f9f-7m9w2\" (UID: \"d47e57de-4b10-40f8-99c4-b127c59c8930\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65448f9f-7m9w2" Jul 13 22:26:41.250349 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:41.250237 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d47e57de-4b10-40f8-99c4-b127c59c8930-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-65448f9f-7m9w2\" (UID: \"d47e57de-4b10-40f8-99c4-b127c59c8930\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65448f9f-7m9w2" Jul 13 22:26:41.250923 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:41.250899 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d47e57de-4b10-40f8-99c4-b127c59c8930-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-65448f9f-7m9w2\" (UID: \"d47e57de-4b10-40f8-99c4-b127c59c8930\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65448f9f-7m9w2" Jul 13 22:26:41.313250 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:41.313217 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65448f9f-7m9w2" Jul 13 22:26:41.448724 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:41.448694 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65448f9f-7m9w2"] Jul 13 22:26:41.450731 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:26:41.450709 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd47e57de_4b10_40f8_99c4_b127c59c8930.slice/crio-33cf97b3ea38bd08945564808ef2ce4e52fd8dfeb2174e94c82dd71d70f958f0 WatchSource:0}: Error finding container 33cf97b3ea38bd08945564808ef2ce4e52fd8dfeb2174e94c82dd71d70f958f0: Status 404 returned error can't find the container with id 33cf97b3ea38bd08945564808ef2ce4e52fd8dfeb2174e94c82dd71d70f958f0 Jul 13 22:26:41.841966 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:41.841929 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65448f9f-7m9w2" event={"ID":"d47e57de-4b10-40f8-99c4-b127c59c8930","Type":"ContainerStarted","Data":"45c0354c601dcdeb4113e9864d5c75f01a6c36c5fc68e83bb13d4d21a8aafdb8"} Jul 13 22:26:41.842135 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:41.841986 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65448f9f-7m9w2" event={"ID":"d47e57de-4b10-40f8-99c4-b127c59c8930","Type":"ContainerStarted","Data":"33cf97b3ea38bd08945564808ef2ce4e52fd8dfeb2174e94c82dd71d70f958f0"} Jul 13 22:26:41.858390 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:41.858340 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65448f9f-7m9w2" podStartSLOduration=1.8583250310000001 podStartE2EDuration="1.858325031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:26:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:26:41.857419295 +0000 UTC m=+1607.851173589" watchObservedRunningTime="2026-07-13 22:26:41.858325031 +0000 UTC m=+1607.852079323" Jul 13 22:26:42.849685 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:42.849650 2555 generic.go:356] "Generic (PLEG): container finished" podID="d47e57de-4b10-40f8-99c4-b127c59c8930" containerID="45c0354c601dcdeb4113e9864d5c75f01a6c36c5fc68e83bb13d4d21a8aafdb8" exitCode=0 Jul 13 22:26:42.850091 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:42.849740 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65448f9f-7m9w2" event={"ID":"d47e57de-4b10-40f8-99c4-b127c59c8930","Type":"ContainerDied","Data":"45c0354c601dcdeb4113e9864d5c75f01a6c36c5fc68e83bb13d4d21a8aafdb8"} Jul 13 22:26:43.990153 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:43.990126 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65448f9f-7m9w2" Jul 13 22:26:44.079144 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:44.079105 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d47e57de-4b10-40f8-99c4-b127c59c8930-maas-api-service-ca\") pod \"d47e57de-4b10-40f8-99c4-b127c59c8930\" (UID: \"d47e57de-4b10-40f8-99c4-b127c59c8930\") " Jul 13 22:26:44.079508 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:44.079485 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d47e57de-4b10-40f8-99c4-b127c59c8930-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d47e57de-4b10-40f8-99c4-b127c59c8930" (UID: "d47e57de-4b10-40f8-99c4-b127c59c8930"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:26:44.180554 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:44.180517 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d47e57de-4b10-40f8-99c4-b127c59c8930-maas-api-service-ca\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:26:44.863186 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:44.863070 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65448f9f-7m9w2" event={"ID":"d47e57de-4b10-40f8-99c4-b127c59c8930","Type":"ContainerDied","Data":"33cf97b3ea38bd08945564808ef2ce4e52fd8dfeb2174e94c82dd71d70f958f0"} Jul 13 22:26:44.863186 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:44.863089 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-65448f9f-7m9w2" Jul 13 22:26:44.863186 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:44.863118 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33cf97b3ea38bd08945564808ef2ce4e52fd8dfeb2174e94c82dd71d70f958f0" Jul 13 22:26:51.063382 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:51.063346 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b"] Jul 13 22:26:51.063881 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:51.063597 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b" podUID="93867ce3-9701-406e-94cf-6c30c413a3dc" containerName="istio-proxy" containerID="cri-o://4083bb810a429fa2a34781b28906661bcb77e1b3d1062dddb2b5c18f328fa4bf" gracePeriod=30 Jul 13 22:26:56.329860 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:56.329792 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b" Jul 13 22:26:56.395484 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:56.395451 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93867ce3-9701-406e-94cf-6c30c413a3dc-istio-podinfo\") pod \"93867ce3-9701-406e-94cf-6c30c413a3dc\" (UID: \"93867ce3-9701-406e-94cf-6c30c413a3dc\") " Jul 13 22:26:56.395653 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:56.395547 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93867ce3-9701-406e-94cf-6c30c413a3dc-istio-token\") pod \"93867ce3-9701-406e-94cf-6c30c413a3dc\" (UID: \"93867ce3-9701-406e-94cf-6c30c413a3dc\") " Jul 13 22:26:56.395653 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:56.395614 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7kmrk\" (UniqueName: \"kubernetes.io/projected/93867ce3-9701-406e-94cf-6c30c413a3dc-kube-api-access-7kmrk\") pod \"93867ce3-9701-406e-94cf-6c30c413a3dc\" (UID: \"93867ce3-9701-406e-94cf-6c30c413a3dc\") " Jul 13 22:26:56.395787 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:56.395684 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93867ce3-9701-406e-94cf-6c30c413a3dc-istio-envoy\") pod \"93867ce3-9701-406e-94cf-6c30c413a3dc\" (UID: \"93867ce3-9701-406e-94cf-6c30c413a3dc\") " Jul 13 22:26:56.398256 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:56.398229 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/93867ce3-9701-406e-94cf-6c30c413a3dc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "93867ce3-9701-406e-94cf-6c30c413a3dc" (UID: "93867ce3-9701-406e-94cf-6c30c413a3dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 13 22:26:56.398256 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:56.398233 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93867ce3-9701-406e-94cf-6c30c413a3dc-istio-token" (OuterVolumeSpecName: "istio-token") pod "93867ce3-9701-406e-94cf-6c30c413a3dc" (UID: "93867ce3-9701-406e-94cf-6c30c413a3dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:26:56.398456 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:56.398400 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93867ce3-9701-406e-94cf-6c30c413a3dc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "93867ce3-9701-406e-94cf-6c30c413a3dc" (UID: "93867ce3-9701-406e-94cf-6c30c413a3dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:26:56.398517 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:56.398458 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93867ce3-9701-406e-94cf-6c30c413a3dc-kube-api-access-7kmrk" (OuterVolumeSpecName: "kube-api-access-7kmrk") pod "93867ce3-9701-406e-94cf-6c30c413a3dc" (UID: "93867ce3-9701-406e-94cf-6c30c413a3dc"). InnerVolumeSpecName "kube-api-access-7kmrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:26:56.496537 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:56.496499 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93867ce3-9701-406e-94cf-6c30c413a3dc-workload-certs\") pod \"93867ce3-9701-406e-94cf-6c30c413a3dc\" (UID: \"93867ce3-9701-406e-94cf-6c30c413a3dc\") " Jul 13 22:26:56.496537 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:56.496541 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93867ce3-9701-406e-94cf-6c30c413a3dc-workload-socket\") pod \"93867ce3-9701-406e-94cf-6c30c413a3dc\" (UID: \"93867ce3-9701-406e-94cf-6c30c413a3dc\") " Jul 13 22:26:56.496752 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:56.496566 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93867ce3-9701-406e-94cf-6c30c413a3dc-istiod-ca-cert\") pod \"93867ce3-9701-406e-94cf-6c30c413a3dc\" (UID: \"93867ce3-9701-406e-94cf-6c30c413a3dc\") " Jul 13 22:26:56.496752 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:56.496639 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93867ce3-9701-406e-94cf-6c30c413a3dc-istio-data\") pod \"93867ce3-9701-406e-94cf-6c30c413a3dc\" (UID: \"93867ce3-9701-406e-94cf-6c30c413a3dc\") " Jul 13 22:26:56.496752 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:56.496658 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93867ce3-9701-406e-94cf-6c30c413a3dc-credential-socket\") pod \"93867ce3-9701-406e-94cf-6c30c413a3dc\" (UID: \"93867ce3-9701-406e-94cf-6c30c413a3dc\") " Jul 13 22:26:56.496904 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:56.496825 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93867ce3-9701-406e-94cf-6c30c413a3dc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "93867ce3-9701-406e-94cf-6c30c413a3dc" (UID: "93867ce3-9701-406e-94cf-6c30c413a3dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:26:56.496904 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:56.496829 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93867ce3-9701-406e-94cf-6c30c413a3dc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "93867ce3-9701-406e-94cf-6c30c413a3dc" (UID: "93867ce3-9701-406e-94cf-6c30c413a3dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:26:56.497032 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:56.496998 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93867ce3-9701-406e-94cf-6c30c413a3dc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "93867ce3-9701-406e-94cf-6c30c413a3dc" (UID: "93867ce3-9701-406e-94cf-6c30c413a3dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:26:56.497032 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:56.497006 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93867ce3-9701-406e-94cf-6c30c413a3dc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "93867ce3-9701-406e-94cf-6c30c413a3dc" (UID: "93867ce3-9701-406e-94cf-6c30c413a3dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:26:56.497247 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:56.497182 2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93867ce3-9701-406e-94cf-6c30c413a3dc-istio-podinfo\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:26:56.497384 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:56.497252 2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93867ce3-9701-406e-94cf-6c30c413a3dc-istio-token\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:26:56.497384 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:56.497267 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kmrk\" (UniqueName: \"kubernetes.io/projected/93867ce3-9701-406e-94cf-6c30c413a3dc-kube-api-access-7kmrk\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:26:56.497384 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:56.497280 2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93867ce3-9701-406e-94cf-6c30c413a3dc-istio-envoy\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:26:56.497641 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:56.497556 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93867ce3-9701-406e-94cf-6c30c413a3dc-istio-data" (OuterVolumeSpecName: "istio-data") pod "93867ce3-9701-406e-94cf-6c30c413a3dc" (UID: "93867ce3-9701-406e-94cf-6c30c413a3dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:26:56.597701 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:56.597618 2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93867ce3-9701-406e-94cf-6c30c413a3dc-istio-data\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:26:56.597701 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:56.597644 2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93867ce3-9701-406e-94cf-6c30c413a3dc-credential-socket\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:26:56.597701 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:56.597655 2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93867ce3-9701-406e-94cf-6c30c413a3dc-workload-certs\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:26:56.597701 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:56.597665 2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93867ce3-9701-406e-94cf-6c30c413a3dc-workload-socket\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:26:56.597701 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:56.597674 2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93867ce3-9701-406e-94cf-6c30c413a3dc-istiod-ca-cert\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:26:56.931020 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:56.930980 2555 generic.go:356] "Generic (PLEG): container finished" podID="93867ce3-9701-406e-94cf-6c30c413a3dc" containerID="4083bb810a429fa2a34781b28906661bcb77e1b3d1062dddb2b5c18f328fa4bf" exitCode=0 Jul 13 22:26:56.931236 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:56.931053 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b" Jul 13 22:26:56.931236 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:56.931069 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b" event={"ID":"93867ce3-9701-406e-94cf-6c30c413a3dc","Type":"ContainerDied","Data":"4083bb810a429fa2a34781b28906661bcb77e1b3d1062dddb2b5c18f328fa4bf"} Jul 13 22:26:56.931236 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:56.931129 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b" event={"ID":"93867ce3-9701-406e-94cf-6c30c413a3dc","Type":"ContainerDied","Data":"59620a54368757b2a1121301ef9d3c43f084e69df425cf8781452a7e0bee1eff"} Jul 13 22:26:56.931236 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:56.931152 2555 scope.go:117] "RemoveContainer" containerID="4083bb810a429fa2a34781b28906661bcb77e1b3d1062dddb2b5c18f328fa4bf" Jul 13 22:26:56.940510 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:56.940489 2555 scope.go:117] "RemoveContainer" containerID="ef641058c01c317cfea88288b3c388b5d10a4d8d49897506fdc7b28fd62e9643" Jul 13 22:26:56.949380 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:56.949356 2555 scope.go:117] "RemoveContainer" containerID="4083bb810a429fa2a34781b28906661bcb77e1b3d1062dddb2b5c18f328fa4bf" Jul 13 22:26:56.949675 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:26:56.949653 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4083bb810a429fa2a34781b28906661bcb77e1b3d1062dddb2b5c18f328fa4bf\": container with ID starting with 4083bb810a429fa2a34781b28906661bcb77e1b3d1062dddb2b5c18f328fa4bf not found: ID does not exist" containerID="4083bb810a429fa2a34781b28906661bcb77e1b3d1062dddb2b5c18f328fa4bf" Jul 13 22:26:56.949756 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:56.949691 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4083bb810a429fa2a34781b28906661bcb77e1b3d1062dddb2b5c18f328fa4bf"} err="failed to get container status \"4083bb810a429fa2a34781b28906661bcb77e1b3d1062dddb2b5c18f328fa4bf\": rpc error: code = NotFound desc = could not find container \"4083bb810a429fa2a34781b28906661bcb77e1b3d1062dddb2b5c18f328fa4bf\": container with ID starting with 4083bb810a429fa2a34781b28906661bcb77e1b3d1062dddb2b5c18f328fa4bf not found: ID does not exist" Jul 13 22:26:56.949756 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:56.949716 2555 scope.go:117] "RemoveContainer" containerID="ef641058c01c317cfea88288b3c388b5d10a4d8d49897506fdc7b28fd62e9643" Jul 13 22:26:56.950039 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:26:56.950019 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef641058c01c317cfea88288b3c388b5d10a4d8d49897506fdc7b28fd62e9643\": container with ID starting with ef641058c01c317cfea88288b3c388b5d10a4d8d49897506fdc7b28fd62e9643 not found: ID does not exist" containerID="ef641058c01c317cfea88288b3c388b5d10a4d8d49897506fdc7b28fd62e9643" Jul 13 22:26:56.950098 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:56.950051 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef641058c01c317cfea88288b3c388b5d10a4d8d49897506fdc7b28fd62e9643"} err="failed to get container status \"ef641058c01c317cfea88288b3c388b5d10a4d8d49897506fdc7b28fd62e9643\": rpc error: code = NotFound desc = could not find container \"ef641058c01c317cfea88288b3c388b5d10a4d8d49897506fdc7b28fd62e9643\": container with ID starting with ef641058c01c317cfea88288b3c388b5d10a4d8d49897506fdc7b28fd62e9643 not found: ID does not exist" Jul 13 22:26:56.953041 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:56.953017 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b"] Jul 13 22:26:56.957875 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:56.957848 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-65448f9f-gw-openshift-default-799799b7f9-kl89b"] Jul 13 22:26:57.339687 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:57.339606 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s"] Jul 13 22:26:57.340177 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:57.340161 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbd8b1f6-e75c-42af-8ba1-1b7694dea93c" containerName="revoke-keys" Jul 13 22:26:57.340244 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:57.340181 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbd8b1f6-e75c-42af-8ba1-1b7694dea93c" containerName="revoke-keys" Jul 13 22:26:57.340244 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:57.340225 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93867ce3-9701-406e-94cf-6c30c413a3dc" containerName="istio-proxy" Jul 13 22:26:57.340244 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:57.340236 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="93867ce3-9701-406e-94cf-6c30c413a3dc" containerName="istio-proxy" Jul 13 22:26:57.340358 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:57.340250 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d47e57de-4b10-40f8-99c4-b127c59c8930" containerName="revoke-keys" Jul 13 22:26:57.340358 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:57.340258 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d47e57de-4b10-40f8-99c4-b127c59c8930" containerName="revoke-keys" Jul 13 22:26:57.340358 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:57.340277 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93867ce3-9701-406e-94cf-6c30c413a3dc" containerName="istio-proxy" Jul 13 22:26:57.340358 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:57.340286 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="93867ce3-9701-406e-94cf-6c30c413a3dc" containerName="istio-proxy" Jul 13 22:26:57.340476 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:57.340367 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d47e57de-4b10-40f8-99c4-b127c59c8930" containerName="revoke-keys" Jul 13 22:26:57.340476 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:57.340378 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="93867ce3-9701-406e-94cf-6c30c413a3dc" containerName="istio-proxy" Jul 13 22:26:57.340476 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:57.340389 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="93867ce3-9701-406e-94cf-6c30c413a3dc" containerName="istio-proxy" Jul 13 22:26:57.345443 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:57.345416 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s" Jul 13 22:26:57.349257 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:57.349235 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-a399a528-openshift-default-dockercfg-9hg94\"" Jul 13 22:26:57.359093 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:57.359059 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s"] Jul 13 22:26:57.402769 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:57.402734 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dab9b71-f170-46a5-b548-f63120731981-workload-certs\") pod \"e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s\" (UID: \"9dab9b71-f170-46a5-b548-f63120731981\") " pod="openshift-ingress/e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s" Jul 13 22:26:57.402769 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:57.402770 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dab9b71-f170-46a5-b548-f63120731981-istiod-ca-cert\") pod \"e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s\" (UID: \"9dab9b71-f170-46a5-b548-f63120731981\") " pod="openshift-ingress/e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s" Jul 13 22:26:57.403077 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:57.402799 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwcv8\" (UniqueName: \"kubernetes.io/projected/9dab9b71-f170-46a5-b548-f63120731981-kube-api-access-jwcv8\") pod \"e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s\" (UID: \"9dab9b71-f170-46a5-b548-f63120731981\") " pod="openshift-ingress/e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s" Jul 13 22:26:57.403077 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:57.402856 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dab9b71-f170-46a5-b548-f63120731981-istio-data\") pod \"e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s\" (UID: \"9dab9b71-f170-46a5-b548-f63120731981\") " pod="openshift-ingress/e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s" Jul 13 22:26:57.403077 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:57.402929 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dab9b71-f170-46a5-b548-f63120731981-credential-socket\") pod \"e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s\" (UID: \"9dab9b71-f170-46a5-b548-f63120731981\") " pod="openshift-ingress/e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s" Jul 13 22:26:57.403077 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:57.402961 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dab9b71-f170-46a5-b548-f63120731981-istio-podinfo\") pod \"e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s\" (UID: \"9dab9b71-f170-46a5-b548-f63120731981\") " pod="openshift-ingress/e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s" Jul 13 22:26:57.403077 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:57.402990 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dab9b71-f170-46a5-b548-f63120731981-workload-socket\") pod \"e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s\" (UID: \"9dab9b71-f170-46a5-b548-f63120731981\") " pod="openshift-ingress/e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s" Jul 13 22:26:57.403077 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:57.403020 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dab9b71-f170-46a5-b548-f63120731981-istio-token\") pod \"e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s\" (UID: \"9dab9b71-f170-46a5-b548-f63120731981\") " pod="openshift-ingress/e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s" Jul 13 22:26:57.403077 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:57.403050 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dab9b71-f170-46a5-b548-f63120731981-istio-envoy\") pod \"e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s\" (UID: \"9dab9b71-f170-46a5-b548-f63120731981\") " pod="openshift-ingress/e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s" Jul 13 22:26:57.503818 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:57.503786 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwcv8\" (UniqueName: \"kubernetes.io/projected/9dab9b71-f170-46a5-b548-f63120731981-kube-api-access-jwcv8\") pod \"e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s\" (UID: \"9dab9b71-f170-46a5-b548-f63120731981\") " pod="openshift-ingress/e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s" Jul 13 22:26:57.503818 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:57.503827 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dab9b71-f170-46a5-b548-f63120731981-istio-data\") pod \"e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s\" (UID: \"9dab9b71-f170-46a5-b548-f63120731981\") " pod="openshift-ingress/e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s" Jul 13 22:26:57.504079 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:57.503863 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dab9b71-f170-46a5-b548-f63120731981-credential-socket\") pod \"e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s\" (UID: \"9dab9b71-f170-46a5-b548-f63120731981\") " pod="openshift-ingress/e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s" Jul 13 22:26:57.504079 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:57.503904 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dab9b71-f170-46a5-b548-f63120731981-istio-podinfo\") pod \"e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s\" (UID: \"9dab9b71-f170-46a5-b548-f63120731981\") " pod="openshift-ingress/e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s" Jul 13 22:26:57.504079 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:57.503929 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dab9b71-f170-46a5-b548-f63120731981-workload-socket\") pod \"e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s\" (UID: \"9dab9b71-f170-46a5-b548-f63120731981\") " pod="openshift-ingress/e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s" Jul 13 22:26:57.504079 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:57.503975 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dab9b71-f170-46a5-b548-f63120731981-istio-token\") pod \"e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s\" (UID: \"9dab9b71-f170-46a5-b548-f63120731981\") " pod="openshift-ingress/e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s" Jul 13 22:26:57.504079 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:57.504024 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dab9b71-f170-46a5-b548-f63120731981-istio-envoy\") pod \"e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s\" (UID: \"9dab9b71-f170-46a5-b548-f63120731981\") " pod="openshift-ingress/e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s" Jul 13 22:26:57.504079 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:57.504063 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dab9b71-f170-46a5-b548-f63120731981-workload-certs\") pod \"e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s\" (UID: \"9dab9b71-f170-46a5-b548-f63120731981\") " pod="openshift-ingress/e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s" Jul 13 22:26:57.504438 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:57.504084 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dab9b71-f170-46a5-b548-f63120731981-istiod-ca-cert\") pod \"e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s\" (UID: \"9dab9b71-f170-46a5-b548-f63120731981\") " pod="openshift-ingress/e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s" Jul 13 22:26:57.504438 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:57.504308 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dab9b71-f170-46a5-b548-f63120731981-istio-data\") pod \"e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s\" (UID: \"9dab9b71-f170-46a5-b548-f63120731981\") " pod="openshift-ingress/e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s" Jul 13 22:26:57.504438 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:57.504378 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dab9b71-f170-46a5-b548-f63120731981-workload-socket\") pod \"e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s\" (UID: \"9dab9b71-f170-46a5-b548-f63120731981\") " pod="openshift-ingress/e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s" Jul 13 22:26:57.504595 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:57.504490 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dab9b71-f170-46a5-b548-f63120731981-credential-socket\") pod \"e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s\" (UID: \"9dab9b71-f170-46a5-b548-f63120731981\") " pod="openshift-ingress/e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s" Jul 13 22:26:57.505193 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:57.504639 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dab9b71-f170-46a5-b548-f63120731981-workload-certs\") pod \"e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s\" (UID: \"9dab9b71-f170-46a5-b548-f63120731981\") " pod="openshift-ingress/e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s" Jul 13 22:26:57.505193 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:57.504970 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dab9b71-f170-46a5-b548-f63120731981-istiod-ca-cert\") pod \"e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s\" (UID: \"9dab9b71-f170-46a5-b548-f63120731981\") " pod="openshift-ingress/e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s" Jul 13 22:26:57.506653 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:57.506629 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dab9b71-f170-46a5-b548-f63120731981-istio-envoy\") pod \"e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s\" (UID: \"9dab9b71-f170-46a5-b548-f63120731981\") " pod="openshift-ingress/e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s" Jul 13 22:26:57.506844 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:57.506827 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dab9b71-f170-46a5-b548-f63120731981-istio-podinfo\") pod \"e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s\" (UID: \"9dab9b71-f170-46a5-b548-f63120731981\") " pod="openshift-ingress/e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s" Jul 13 22:26:57.511931 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:57.511907 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dab9b71-f170-46a5-b548-f63120731981-istio-token\") pod \"e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s\" (UID: \"9dab9b71-f170-46a5-b548-f63120731981\") " pod="openshift-ingress/e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s" Jul 13 22:26:57.512158 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:57.512140 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwcv8\" (UniqueName: \"kubernetes.io/projected/9dab9b71-f170-46a5-b548-f63120731981-kube-api-access-jwcv8\") pod \"e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s\" (UID: \"9dab9b71-f170-46a5-b548-f63120731981\") " pod="openshift-ingress/e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s" Jul 13 22:26:57.657380 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:57.657347 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s" Jul 13 22:26:57.808307 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:57.808270 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s"] Jul 13 22:26:57.809443 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:26:57.809406 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dab9b71_f170_46a5_b548_f63120731981.slice/crio-39b5d119c65fbf3269202e0b499627cf946b17acbe72abf84dacd23f49a33cb2 WatchSource:0}: Error finding container 39b5d119c65fbf3269202e0b499627cf946b17acbe72abf84dacd23f49a33cb2: Status 404 returned error can't find the container with id 39b5d119c65fbf3269202e0b499627cf946b17acbe72abf84dacd23f49a33cb2 Jul 13 22:26:57.812106 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:57.812072 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:26:57.812174 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:57.812158 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:26:57.812270 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:57.812229 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:26:57.940422 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:57.940384 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s" event={"ID":"9dab9b71-f170-46a5-b548-f63120731981","Type":"ContainerStarted","Data":"9176f2f1493fe56d1f1ffde68979ff5f7c84944b090d943d84924d38b766b7cb"} Jul 13 22:26:57.940630 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:57.940448 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s" event={"ID":"9dab9b71-f170-46a5-b548-f63120731981","Type":"ContainerStarted","Data":"39b5d119c65fbf3269202e0b499627cf946b17acbe72abf84dacd23f49a33cb2"} Jul 13 22:26:57.960354 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:57.960291 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s" podStartSLOduration=0.960269499 podStartE2EDuration="960.269499ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:26:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:26:57.959757437 +0000 UTC m=+1623.953511731" watchObservedRunningTime="2026-07-13 22:26:57.960269499 +0000 UTC m=+1623.954023795" Jul 13 22:26:58.560287 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:58.560242 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93867ce3-9701-406e-94cf-6c30c413a3dc" path="/var/lib/kubelet/pods/93867ce3-9701-406e-94cf-6c30c413a3dc/volumes" Jul 13 22:26:58.658501 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:58.658465 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s" Jul 13 22:26:58.664025 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:58.663997 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s" Jul 13 22:26:58.947802 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:58.947770 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s" Jul 13 22:26:58.948872 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:26:58.948850 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s" Jul 13 22:27:03.563074 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:03.563040 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a399a528-qv5d4"] Jul 13 22:27:03.567776 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:03.567756 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a399a528-qv5d4" Jul 13 22:27:03.571629 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:03.571581 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k5z5d\"" Jul 13 22:27:03.571969 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:03.571844 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 13 22:27:03.574913 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:03.574885 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a399a528-qv5d4"] Jul 13 22:27:03.661449 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:03.661409 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6713a194-3062-4644-91df-779757113d8a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a399a528-qv5d4\" (UID: \"6713a194-3062-4644-91df-779757113d8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a399a528-qv5d4" Jul 13 22:27:03.762662 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:03.762621 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6713a194-3062-4644-91df-779757113d8a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a399a528-qv5d4\" (UID: \"6713a194-3062-4644-91df-779757113d8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a399a528-qv5d4" Jul 13 22:27:03.763274 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:03.763253 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6713a194-3062-4644-91df-779757113d8a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a399a528-qv5d4\" (UID: \"6713a194-3062-4644-91df-779757113d8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a399a528-qv5d4" Jul 13 22:27:03.880289 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:03.880261 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a399a528-qv5d4" Jul 13 22:27:04.040313 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:04.040285 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a399a528-qv5d4"] Jul 13 22:27:04.041752 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:27:04.041722 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6713a194_3062_4644_91df_779757113d8a.slice/crio-e79876feee3aab260072343b5f9c1b4925c82816c470f450384c5856e56fc775 WatchSource:0}: Error finding container e79876feee3aab260072343b5f9c1b4925c82816c470f450384c5856e56fc775: Status 404 returned error can't find the container with id e79876feee3aab260072343b5f9c1b4925c82816c470f450384c5856e56fc775 Jul 13 22:27:04.983478 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:04.983375 2555 generic.go:356] "Generic (PLEG): container finished" podID="6713a194-3062-4644-91df-779757113d8a" containerID="7359d927d0fa50f96606962d38fda553f9cdf32e0c3a8bf98eed18a99543a158" exitCode=7 Jul 13 22:27:04.983478 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:04.983444 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a399a528-qv5d4" event={"ID":"6713a194-3062-4644-91df-779757113d8a","Type":"ContainerDied","Data":"7359d927d0fa50f96606962d38fda553f9cdf32e0c3a8bf98eed18a99543a158"} Jul 13 22:27:04.983881 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:04.983500 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a399a528-qv5d4" event={"ID":"6713a194-3062-4644-91df-779757113d8a","Type":"ContainerStarted","Data":"e79876feee3aab260072343b5f9c1b4925c82816c470f450384c5856e56fc775"} Jul 13 22:27:04.983881 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:04.983610 2555 scope.go:117] "RemoveContainer" containerID="7359d927d0fa50f96606962d38fda553f9cdf32e0c3a8bf98eed18a99543a158" Jul 13 22:27:05.991525 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:05.991484 2555 generic.go:356] "Generic (PLEG): container finished" podID="6713a194-3062-4644-91df-779757113d8a" containerID="934d7a2e463e69ab059763456bec2dbbb528bbd496856b2287f21e3bad0bbcc8" exitCode=7 Jul 13 22:27:05.992187 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:05.991581 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a399a528-qv5d4" event={"ID":"6713a194-3062-4644-91df-779757113d8a","Type":"ContainerDied","Data":"934d7a2e463e69ab059763456bec2dbbb528bbd496856b2287f21e3bad0bbcc8"} Jul 13 22:27:05.992187 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:05.991679 2555 scope.go:117] "RemoveContainer" containerID="7359d927d0fa50f96606962d38fda553f9cdf32e0c3a8bf98eed18a99543a158" Jul 13 22:27:05.992187 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:05.991945 2555 scope.go:117] "RemoveContainer" containerID="934d7a2e463e69ab059763456bec2dbbb528bbd496856b2287f21e3bad0bbcc8" Jul 13 22:27:05.992484 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:27:05.992188 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a399a528-qv5d4_odh-ai-gateway-infra(6713a194-3062-4644-91df-779757113d8a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a399a528-qv5d4" podUID="6713a194-3062-4644-91df-779757113d8a" Jul 13 22:27:07.007033 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:07.007003 2555 scope.go:117] "RemoveContainer" containerID="934d7a2e463e69ab059763456bec2dbbb528bbd496856b2287f21e3bad0bbcc8" Jul 13 22:27:07.007429 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:27:07.007217 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a399a528-qv5d4_odh-ai-gateway-infra(6713a194-3062-4644-91df-779757113d8a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a399a528-qv5d4" podUID="6713a194-3062-4644-91df-779757113d8a" Jul 13 22:27:14.167863 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:14.167828 2555 patch_prober.go:28] interesting pod/e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": read tcp 10.134.0.2:56750->10.134.0.55:15021: read: connection reset by peer" start-of-body= Jul 13 22:27:14.168292 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:14.167898 2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s" podUID="9dab9b71-f170-46a5-b548-f63120731981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": read tcp 10.134.0.2:56750->10.134.0.55:15021: read: connection reset by peer" Jul 13 22:27:15.059033 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:15.058996 2555 generic.go:356] "Generic (PLEG): container finished" podID="9dab9b71-f170-46a5-b548-f63120731981" containerID="9176f2f1493fe56d1f1ffde68979ff5f7c84944b090d943d84924d38b766b7cb" exitCode=0 Jul 13 22:27:15.059245 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:15.059064 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s" event={"ID":"9dab9b71-f170-46a5-b548-f63120731981","Type":"ContainerDied","Data":"9176f2f1493fe56d1f1ffde68979ff5f7c84944b090d943d84924d38b766b7cb"} Jul 13 22:27:15.059486 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:15.059472 2555 scope.go:117] "RemoveContainer" containerID="9176f2f1493fe56d1f1ffde68979ff5f7c84944b090d943d84924d38b766b7cb" Jul 13 22:27:15.060440 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:15.060408 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:27:15.060606 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:15.060479 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:27:15.060606 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:15.060536 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:27:16.066453 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:16.066412 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s" event={"ID":"9dab9b71-f170-46a5-b548-f63120731981","Type":"ContainerStarted","Data":"98c1ac726d7f59d89c33ef4b916e1810d7ddae60316bad6e0c6e128fb429c43e"} Jul 13 22:27:16.658570 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:16.658533 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s" Jul 13 22:27:16.658791 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:16.658761 2555 patch_prober.go:28] interesting pod/e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 13 22:27:16.658866 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:16.658815 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s" podUID="9dab9b71-f170-46a5-b548-f63120731981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 13 22:27:17.658599 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:17.658559 2555 patch_prober.go:28] interesting pod/e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 13 22:27:17.658997 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:17.658623 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s" podUID="9dab9b71-f170-46a5-b548-f63120731981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 13 22:27:18.658669 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:18.658629 2555 patch_prober.go:28] interesting pod/e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 13 22:27:18.659060 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:18.658690 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s" podUID="9dab9b71-f170-46a5-b548-f63120731981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 13 22:27:19.553774 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:19.553737 2555 scope.go:117] "RemoveContainer" containerID="934d7a2e463e69ab059763456bec2dbbb528bbd496856b2287f21e3bad0bbcc8" Jul 13 22:27:19.658130 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:19.658089 2555 patch_prober.go:28] interesting pod/e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 13 22:27:19.658356 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:19.658158 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s" podUID="9dab9b71-f170-46a5-b548-f63120731981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 13 22:27:20.090998 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:20.090906 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a399a528-qv5d4" event={"ID":"6713a194-3062-4644-91df-779757113d8a","Type":"ContainerStarted","Data":"ce0a64eef6fb398912265571744000af7a58e2db4878eb967d4e41d6c978ff5e"} Jul 13 22:27:20.110257 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:20.110155 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a399a528-qv5d4" podStartSLOduration=17.11012846 podStartE2EDuration="17.11012846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:27:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:27:20.106605691 +0000 UTC m=+1646.100360004" watchObservedRunningTime="2026-07-13 22:27:20.11012846 +0000 UTC m=+1646.103882822" Jul 13 22:27:20.661514 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:20.661435 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s" Jul 13 22:27:20.661725 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:20.661708 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s" Jul 13 22:27:20.662571 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:20.662552 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s" Jul 13 22:27:21.098380 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:21.098296 2555 generic.go:356] "Generic (PLEG): container finished" podID="6713a194-3062-4644-91df-779757113d8a" containerID="ce0a64eef6fb398912265571744000af7a58e2db4878eb967d4e41d6c978ff5e" exitCode=0 Jul 13 22:27:21.098788 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:21.098377 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a399a528-qv5d4" event={"ID":"6713a194-3062-4644-91df-779757113d8a","Type":"ContainerDied","Data":"ce0a64eef6fb398912265571744000af7a58e2db4878eb967d4e41d6c978ff5e"} Jul 13 22:27:21.098788 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:21.098439 2555 scope.go:117] "RemoveContainer" containerID="934d7a2e463e69ab059763456bec2dbbb528bbd496856b2287f21e3bad0bbcc8" Jul 13 22:27:21.126062 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:21.126039 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a399a528-qv5d4"] Jul 13 22:27:22.232562 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:22.232533 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a399a528-qv5d4" Jul 13 22:27:22.342558 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:22.342523 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6713a194-3062-4644-91df-779757113d8a-maas-api-service-ca\") pod \"6713a194-3062-4644-91df-779757113d8a\" (UID: \"6713a194-3062-4644-91df-779757113d8a\") " Jul 13 22:27:22.342898 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:22.342877 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6713a194-3062-4644-91df-779757113d8a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6713a194-3062-4644-91df-779757113d8a" (UID: "6713a194-3062-4644-91df-779757113d8a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:27:22.443481 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:22.443445 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6713a194-3062-4644-91df-779757113d8a-maas-api-service-ca\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:27:23.112190 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:23.112149 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a399a528-qv5d4" Jul 13 22:27:23.112410 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:23.112195 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a399a528-qv5d4" event={"ID":"6713a194-3062-4644-91df-779757113d8a","Type":"ContainerDied","Data":"e79876feee3aab260072343b5f9c1b4925c82816c470f450384c5856e56fc775"} Jul 13 22:27:23.112410 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:23.112274 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e79876feee3aab260072343b5f9c1b4925c82816c470f450384c5856e56fc775" Jul 13 22:27:23.151151 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:23.151113 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a399a528-qv5d4"] Jul 13 22:27:23.153451 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:23.153427 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a399a528-qv5d4"] Jul 13 22:27:24.559464 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:24.559432 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6713a194-3062-4644-91df-779757113d8a" path="/var/lib/kubelet/pods/6713a194-3062-4644-91df-779757113d8a/volumes" Jul 13 22:27:53.618550 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:53.618517 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a399a528-mnt2q"] Jul 13 22:27:53.619069 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:53.618872 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6713a194-3062-4644-91df-779757113d8a" containerName="revoke-keys" Jul 13 22:27:53.619069 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:53.618883 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6713a194-3062-4644-91df-779757113d8a" containerName="revoke-keys" Jul 13 22:27:53.619069 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:53.618896 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6713a194-3062-4644-91df-779757113d8a" containerName="revoke-keys" Jul 13 22:27:53.619069 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:53.618901 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6713a194-3062-4644-91df-779757113d8a" containerName="revoke-keys" Jul 13 22:27:53.619069 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:53.618976 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6713a194-3062-4644-91df-779757113d8a" containerName="revoke-keys" Jul 13 22:27:53.619069 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:53.618988 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6713a194-3062-4644-91df-779757113d8a" containerName="revoke-keys" Jul 13 22:27:53.622473 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:53.622454 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a399a528-mnt2q" Jul 13 22:27:53.625184 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:53.625165 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 13 22:27:53.625567 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:53.625549 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k5z5d\"" Jul 13 22:27:53.631298 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:53.631278 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a399a528-mnt2q"] Jul 13 22:27:53.708456 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:53.708426 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/771d2883-dbbb-4d75-ace3-a6b36abb9e8e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a399a528-mnt2q\" (UID: \"771d2883-dbbb-4d75-ace3-a6b36abb9e8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a399a528-mnt2q" Jul 13 22:27:53.809047 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:53.809004 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/771d2883-dbbb-4d75-ace3-a6b36abb9e8e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a399a528-mnt2q\" (UID: \"771d2883-dbbb-4d75-ace3-a6b36abb9e8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a399a528-mnt2q" Jul 13 22:27:53.809664 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:53.809642 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/771d2883-dbbb-4d75-ace3-a6b36abb9e8e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a399a528-mnt2q\" (UID: \"771d2883-dbbb-4d75-ace3-a6b36abb9e8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a399a528-mnt2q" Jul 13 22:27:53.934441 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:53.934393 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a399a528-mnt2q" Jul 13 22:27:54.076505 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:54.076478 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a399a528-mnt2q"] Jul 13 22:27:54.078014 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:27:54.077986 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod771d2883_dbbb_4d75_ace3_a6b36abb9e8e.slice/crio-cf02eb2f6a36cf7da6d9f915b69ba545bf49d98520ccae7eabca030c65347a1a WatchSource:0}: Error finding container cf02eb2f6a36cf7da6d9f915b69ba545bf49d98520ccae7eabca030c65347a1a: Status 404 returned error can't find the container with id cf02eb2f6a36cf7da6d9f915b69ba545bf49d98520ccae7eabca030c65347a1a Jul 13 22:27:54.288167 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:54.288081 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a399a528-mnt2q" event={"ID":"771d2883-dbbb-4d75-ace3-a6b36abb9e8e","Type":"ContainerStarted","Data":"cf02eb2f6a36cf7da6d9f915b69ba545bf49d98520ccae7eabca030c65347a1a"} Jul 13 22:27:55.294837 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:55.294803 2555 generic.go:356] "Generic (PLEG): container finished" podID="771d2883-dbbb-4d75-ace3-a6b36abb9e8e" containerID="ec2aa7d9b006c3ef29cbecc88c86154a165e09b3e4d095de2a47443c5cf89104" exitCode=0 Jul 13 22:27:55.295257 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:55.294895 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a399a528-mnt2q" event={"ID":"771d2883-dbbb-4d75-ace3-a6b36abb9e8e","Type":"ContainerDied","Data":"ec2aa7d9b006c3ef29cbecc88c86154a165e09b3e4d095de2a47443c5cf89104"} Jul 13 22:27:56.434860 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:56.434832 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a399a528-mnt2q" Jul 13 22:27:56.531983 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:56.531928 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/771d2883-dbbb-4d75-ace3-a6b36abb9e8e-maas-api-service-ca\") pod \"771d2883-dbbb-4d75-ace3-a6b36abb9e8e\" (UID: \"771d2883-dbbb-4d75-ace3-a6b36abb9e8e\") " Jul 13 22:27:56.532313 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:56.532290 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/771d2883-dbbb-4d75-ace3-a6b36abb9e8e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "771d2883-dbbb-4d75-ace3-a6b36abb9e8e" (UID: "771d2883-dbbb-4d75-ace3-a6b36abb9e8e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:27:56.633213 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:56.633154 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/771d2883-dbbb-4d75-ace3-a6b36abb9e8e-maas-api-service-ca\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:27:57.307493 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:57.307460 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a399a528-mnt2q" Jul 13 22:27:57.307668 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:57.307465 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a399a528-mnt2q" event={"ID":"771d2883-dbbb-4d75-ace3-a6b36abb9e8e","Type":"ContainerDied","Data":"cf02eb2f6a36cf7da6d9f915b69ba545bf49d98520ccae7eabca030c65347a1a"} Jul 13 22:27:57.307668 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:27:57.307592 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf02eb2f6a36cf7da6d9f915b69ba545bf49d98520ccae7eabca030c65347a1a" Jul 13 22:28:19.702092 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:19.701995 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s"] Jul 13 22:28:19.702583 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:19.702261 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s" podUID="9dab9b71-f170-46a5-b548-f63120731981" containerName="istio-proxy" containerID="cri-o://98c1ac726d7f59d89c33ef4b916e1810d7ddae60316bad6e0c6e128fb429c43e" gracePeriod=30 Jul 13 22:28:20.019114 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:20.019026 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-0081b233-openshift-default-5885f55d47-2wchr"] Jul 13 22:28:20.019699 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:20.019668 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6713a194-3062-4644-91df-779757113d8a" containerName="revoke-keys" Jul 13 22:28:20.019699 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:20.019695 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6713a194-3062-4644-91df-779757113d8a" containerName="revoke-keys" Jul 13 22:28:20.019845 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:20.019709 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="771d2883-dbbb-4d75-ace3-a6b36abb9e8e" containerName="revoke-keys" Jul 13 22:28:20.019845 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:20.019718 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="771d2883-dbbb-4d75-ace3-a6b36abb9e8e" containerName="revoke-keys" Jul 13 22:28:20.019845 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:20.019833 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6713a194-3062-4644-91df-779757113d8a" containerName="revoke-keys" Jul 13 22:28:20.019951 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:20.019852 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="771d2883-dbbb-4d75-ace3-a6b36abb9e8e" containerName="revoke-keys" Jul 13 22:28:20.023432 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:20.023412 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0081b233-openshift-default-5885f55d47-2wchr" Jul 13 22:28:20.026960 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:20.026942 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-0081b233-openshift-default-dockercfg-wl4v4\"" Jul 13 22:28:20.036776 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:20.036748 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-0081b233-openshift-default-5885f55d47-2wchr"] Jul 13 22:28:20.151518 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:20.151468 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4719b18b-0095-49a1-8712-e2a5616721cb-workload-socket\") pod \"e2e-derive-0081b233-openshift-default-5885f55d47-2wchr\" (UID: \"4719b18b-0095-49a1-8712-e2a5616721cb\") " pod="openshift-ingress/e2e-derive-0081b233-openshift-default-5885f55d47-2wchr" Jul 13 22:28:20.151518 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:20.151528 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4719b18b-0095-49a1-8712-e2a5616721cb-workload-certs\") pod \"e2e-derive-0081b233-openshift-default-5885f55d47-2wchr\" (UID: \"4719b18b-0095-49a1-8712-e2a5616721cb\") " pod="openshift-ingress/e2e-derive-0081b233-openshift-default-5885f55d47-2wchr" Jul 13 22:28:20.151741 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:20.151624 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4719b18b-0095-49a1-8712-e2a5616721cb-istio-podinfo\") pod \"e2e-derive-0081b233-openshift-default-5885f55d47-2wchr\" (UID: \"4719b18b-0095-49a1-8712-e2a5616721cb\") " pod="openshift-ingress/e2e-derive-0081b233-openshift-default-5885f55d47-2wchr" Jul 13 22:28:20.151741 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:20.151662 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4719b18b-0095-49a1-8712-e2a5616721cb-istio-envoy\") pod \"e2e-derive-0081b233-openshift-default-5885f55d47-2wchr\" (UID: \"4719b18b-0095-49a1-8712-e2a5616721cb\") " pod="openshift-ingress/e2e-derive-0081b233-openshift-default-5885f55d47-2wchr" Jul 13 22:28:20.151741 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:20.151685 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4719b18b-0095-49a1-8712-e2a5616721cb-istio-token\") pod \"e2e-derive-0081b233-openshift-default-5885f55d47-2wchr\" (UID: \"4719b18b-0095-49a1-8712-e2a5616721cb\") " pod="openshift-ingress/e2e-derive-0081b233-openshift-default-5885f55d47-2wchr" Jul 13 22:28:20.151935 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:20.151760 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4719b18b-0095-49a1-8712-e2a5616721cb-istiod-ca-cert\") pod \"e2e-derive-0081b233-openshift-default-5885f55d47-2wchr\" (UID: \"4719b18b-0095-49a1-8712-e2a5616721cb\") " pod="openshift-ingress/e2e-derive-0081b233-openshift-default-5885f55d47-2wchr" Jul 13 22:28:20.151935 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:20.151797 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4719b18b-0095-49a1-8712-e2a5616721cb-istio-data\") pod \"e2e-derive-0081b233-openshift-default-5885f55d47-2wchr\" (UID: \"4719b18b-0095-49a1-8712-e2a5616721cb\") " pod="openshift-ingress/e2e-derive-0081b233-openshift-default-5885f55d47-2wchr" Jul 13 22:28:20.151935 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:20.151813 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hdbw\" (UniqueName: \"kubernetes.io/projected/4719b18b-0095-49a1-8712-e2a5616721cb-kube-api-access-9hdbw\") pod \"e2e-derive-0081b233-openshift-default-5885f55d47-2wchr\" (UID: \"4719b18b-0095-49a1-8712-e2a5616721cb\") " pod="openshift-ingress/e2e-derive-0081b233-openshift-default-5885f55d47-2wchr" Jul 13 22:28:20.151935 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:20.151863 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4719b18b-0095-49a1-8712-e2a5616721cb-credential-socket\") pod \"e2e-derive-0081b233-openshift-default-5885f55d47-2wchr\" (UID: \"4719b18b-0095-49a1-8712-e2a5616721cb\") " pod="openshift-ingress/e2e-derive-0081b233-openshift-default-5885f55d47-2wchr" Jul 13 22:28:20.253108 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:20.253056 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4719b18b-0095-49a1-8712-e2a5616721cb-istio-data\") pod \"e2e-derive-0081b233-openshift-default-5885f55d47-2wchr\" (UID: \"4719b18b-0095-49a1-8712-e2a5616721cb\") " pod="openshift-ingress/e2e-derive-0081b233-openshift-default-5885f55d47-2wchr" Jul 13 22:28:20.253108 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:20.253114 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hdbw\" (UniqueName: \"kubernetes.io/projected/4719b18b-0095-49a1-8712-e2a5616721cb-kube-api-access-9hdbw\") pod \"e2e-derive-0081b233-openshift-default-5885f55d47-2wchr\" (UID: \"4719b18b-0095-49a1-8712-e2a5616721cb\") " pod="openshift-ingress/e2e-derive-0081b233-openshift-default-5885f55d47-2wchr" Jul 13 22:28:20.253391 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:20.253174 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4719b18b-0095-49a1-8712-e2a5616721cb-credential-socket\") pod \"e2e-derive-0081b233-openshift-default-5885f55d47-2wchr\" (UID: \"4719b18b-0095-49a1-8712-e2a5616721cb\") " pod="openshift-ingress/e2e-derive-0081b233-openshift-default-5885f55d47-2wchr" Jul 13 22:28:20.253391 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:20.253248 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4719b18b-0095-49a1-8712-e2a5616721cb-workload-socket\") pod \"e2e-derive-0081b233-openshift-default-5885f55d47-2wchr\" (UID: \"4719b18b-0095-49a1-8712-e2a5616721cb\") " pod="openshift-ingress/e2e-derive-0081b233-openshift-default-5885f55d47-2wchr" Jul 13 22:28:20.253391 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:20.253287 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4719b18b-0095-49a1-8712-e2a5616721cb-workload-certs\") pod \"e2e-derive-0081b233-openshift-default-5885f55d47-2wchr\" (UID: \"4719b18b-0095-49a1-8712-e2a5616721cb\") " pod="openshift-ingress/e2e-derive-0081b233-openshift-default-5885f55d47-2wchr" Jul 13 22:28:20.253391 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:20.253347 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4719b18b-0095-49a1-8712-e2a5616721cb-istio-podinfo\") pod \"e2e-derive-0081b233-openshift-default-5885f55d47-2wchr\" (UID: \"4719b18b-0095-49a1-8712-e2a5616721cb\") " pod="openshift-ingress/e2e-derive-0081b233-openshift-default-5885f55d47-2wchr" Jul 13 22:28:20.253391 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:20.253379 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4719b18b-0095-49a1-8712-e2a5616721cb-istio-envoy\") pod \"e2e-derive-0081b233-openshift-default-5885f55d47-2wchr\" (UID: \"4719b18b-0095-49a1-8712-e2a5616721cb\") " pod="openshift-ingress/e2e-derive-0081b233-openshift-default-5885f55d47-2wchr" Jul 13 22:28:20.253649 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:20.253407 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4719b18b-0095-49a1-8712-e2a5616721cb-istio-token\") pod \"e2e-derive-0081b233-openshift-default-5885f55d47-2wchr\" (UID: \"4719b18b-0095-49a1-8712-e2a5616721cb\") " pod="openshift-ingress/e2e-derive-0081b233-openshift-default-5885f55d47-2wchr" Jul 13 22:28:20.253649 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:20.253445 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4719b18b-0095-49a1-8712-e2a5616721cb-istiod-ca-cert\") pod \"e2e-derive-0081b233-openshift-default-5885f55d47-2wchr\" (UID: \"4719b18b-0095-49a1-8712-e2a5616721cb\") " pod="openshift-ingress/e2e-derive-0081b233-openshift-default-5885f55d47-2wchr" Jul 13 22:28:20.253649 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:20.253560 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4719b18b-0095-49a1-8712-e2a5616721cb-istio-data\") pod \"e2e-derive-0081b233-openshift-default-5885f55d47-2wchr\" (UID: \"4719b18b-0095-49a1-8712-e2a5616721cb\") " pod="openshift-ingress/e2e-derive-0081b233-openshift-default-5885f55d47-2wchr" Jul 13 22:28:20.253784 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:20.253695 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4719b18b-0095-49a1-8712-e2a5616721cb-credential-socket\") pod \"e2e-derive-0081b233-openshift-default-5885f55d47-2wchr\" (UID: \"4719b18b-0095-49a1-8712-e2a5616721cb\") " pod="openshift-ingress/e2e-derive-0081b233-openshift-default-5885f55d47-2wchr" Jul 13 22:28:20.253784 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:20.253754 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4719b18b-0095-49a1-8712-e2a5616721cb-workload-certs\") pod \"e2e-derive-0081b233-openshift-default-5885f55d47-2wchr\" (UID: \"4719b18b-0095-49a1-8712-e2a5616721cb\") " pod="openshift-ingress/e2e-derive-0081b233-openshift-default-5885f55d47-2wchr" Jul 13 22:28:20.253919 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:20.253894 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4719b18b-0095-49a1-8712-e2a5616721cb-workload-socket\") pod \"e2e-derive-0081b233-openshift-default-5885f55d47-2wchr\" (UID: \"4719b18b-0095-49a1-8712-e2a5616721cb\") " pod="openshift-ingress/e2e-derive-0081b233-openshift-default-5885f55d47-2wchr" Jul 13 22:28:20.254090 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:20.254071 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4719b18b-0095-49a1-8712-e2a5616721cb-istiod-ca-cert\") pod \"e2e-derive-0081b233-openshift-default-5885f55d47-2wchr\" (UID: \"4719b18b-0095-49a1-8712-e2a5616721cb\") " pod="openshift-ingress/e2e-derive-0081b233-openshift-default-5885f55d47-2wchr" Jul 13 22:28:20.255845 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:20.255825 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4719b18b-0095-49a1-8712-e2a5616721cb-istio-envoy\") pod \"e2e-derive-0081b233-openshift-default-5885f55d47-2wchr\" (UID: \"4719b18b-0095-49a1-8712-e2a5616721cb\") " pod="openshift-ingress/e2e-derive-0081b233-openshift-default-5885f55d47-2wchr" Jul 13 22:28:20.255977 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:20.255958 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4719b18b-0095-49a1-8712-e2a5616721cb-istio-podinfo\") pod \"e2e-derive-0081b233-openshift-default-5885f55d47-2wchr\" (UID: \"4719b18b-0095-49a1-8712-e2a5616721cb\") " pod="openshift-ingress/e2e-derive-0081b233-openshift-default-5885f55d47-2wchr" Jul 13 22:28:20.266712 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:20.266690 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4719b18b-0095-49a1-8712-e2a5616721cb-istio-token\") pod \"e2e-derive-0081b233-openshift-default-5885f55d47-2wchr\" (UID: \"4719b18b-0095-49a1-8712-e2a5616721cb\") " pod="openshift-ingress/e2e-derive-0081b233-openshift-default-5885f55d47-2wchr" Jul 13 22:28:20.267065 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:20.267042 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hdbw\" (UniqueName: \"kubernetes.io/projected/4719b18b-0095-49a1-8712-e2a5616721cb-kube-api-access-9hdbw\") pod \"e2e-derive-0081b233-openshift-default-5885f55d47-2wchr\" (UID: \"4719b18b-0095-49a1-8712-e2a5616721cb\") " pod="openshift-ingress/e2e-derive-0081b233-openshift-default-5885f55d47-2wchr" Jul 13 22:28:20.333990 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:20.333898 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0081b233-openshift-default-5885f55d47-2wchr" Jul 13 22:28:20.486544 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:20.486516 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-0081b233-openshift-default-5885f55d47-2wchr"] Jul 13 22:28:20.487899 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:28:20.487873 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4719b18b_0095_49a1_8712_e2a5616721cb.slice/crio-d1bcfe1b0e5732e53b71f0234badd44dda7e7801746f41e7101e3439ed25e011 WatchSource:0}: Error finding container d1bcfe1b0e5732e53b71f0234badd44dda7e7801746f41e7101e3439ed25e011: Status 404 returned error can't find the container with id d1bcfe1b0e5732e53b71f0234badd44dda7e7801746f41e7101e3439ed25e011 Jul 13 22:28:20.490007 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:20.489977 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:28:20.490074 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:20.490061 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:28:20.490116 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:20.490093 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:28:20.664270 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:20.664194 2555 patch_prober.go:28] interesting pod/e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 13 22:28:20.664711 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:20.664291 2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s" podUID="9dab9b71-f170-46a5-b548-f63120731981" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 13 22:28:21.446067 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:21.446031 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0081b233-openshift-default-5885f55d47-2wchr" event={"ID":"4719b18b-0095-49a1-8712-e2a5616721cb","Type":"ContainerStarted","Data":"4aa5d7f5534c4263e2aca046408b6f9f69e505ad82c6dd03e60798ee07a5cd65"} Jul 13 22:28:21.446517 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:21.446087 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0081b233-openshift-default-5885f55d47-2wchr" event={"ID":"4719b18b-0095-49a1-8712-e2a5616721cb","Type":"ContainerStarted","Data":"d1bcfe1b0e5732e53b71f0234badd44dda7e7801746f41e7101e3439ed25e011"} Jul 13 22:28:21.478122 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:21.478063 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-0081b233-openshift-default-5885f55d47-2wchr" podStartSLOduration=2.478044009 podStartE2EDuration="2.478044009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:28:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:28:21.47535819 +0000 UTC m=+1707.469112504" watchObservedRunningTime="2026-07-13 22:28:21.478044009 +0000 UTC m=+1707.471798303" Jul 13 22:28:22.334220 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:22.334159 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-0081b233-openshift-default-5885f55d47-2wchr" Jul 13 22:28:22.339089 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:22.339065 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-0081b233-openshift-default-5885f55d47-2wchr" Jul 13 22:28:22.452632 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:22.452603 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-0081b233-openshift-default-5885f55d47-2wchr" Jul 13 22:28:22.453663 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:22.453642 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-0081b233-openshift-default-5885f55d47-2wchr" Jul 13 22:28:24.954550 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:24.954525 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s" Jul 13 22:28:25.099458 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:25.099426 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dab9b71-f170-46a5-b548-f63120731981-istio-podinfo\") pod \"9dab9b71-f170-46a5-b548-f63120731981\" (UID: \"9dab9b71-f170-46a5-b548-f63120731981\") " Jul 13 22:28:25.099634 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:25.099469 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dab9b71-f170-46a5-b548-f63120731981-credential-socket\") pod \"9dab9b71-f170-46a5-b548-f63120731981\" (UID: \"9dab9b71-f170-46a5-b548-f63120731981\") " Jul 13 22:28:25.099634 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:25.099516 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dab9b71-f170-46a5-b548-f63120731981-istio-envoy\") pod \"9dab9b71-f170-46a5-b548-f63120731981\" (UID: \"9dab9b71-f170-46a5-b548-f63120731981\") " Jul 13 22:28:25.099634 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:25.099564 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jwcv8\" (UniqueName: \"kubernetes.io/projected/9dab9b71-f170-46a5-b548-f63120731981-kube-api-access-jwcv8\") pod \"9dab9b71-f170-46a5-b548-f63120731981\" (UID: \"9dab9b71-f170-46a5-b548-f63120731981\") " Jul 13 22:28:25.099634 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:25.099600 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dab9b71-f170-46a5-b548-f63120731981-workload-certs\") pod \"9dab9b71-f170-46a5-b548-f63120731981\" (UID: \"9dab9b71-f170-46a5-b548-f63120731981\") " Jul 13 22:28:25.099634 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:25.099624 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dab9b71-f170-46a5-b548-f63120731981-istio-token\") pod \"9dab9b71-f170-46a5-b548-f63120731981\" (UID: \"9dab9b71-f170-46a5-b548-f63120731981\") " Jul 13 22:28:25.099892 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:25.099656 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dab9b71-f170-46a5-b548-f63120731981-istiod-ca-cert\") pod \"9dab9b71-f170-46a5-b548-f63120731981\" (UID: \"9dab9b71-f170-46a5-b548-f63120731981\") " Jul 13 22:28:25.099892 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:25.099695 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dab9b71-f170-46a5-b548-f63120731981-workload-socket\") pod \"9dab9b71-f170-46a5-b548-f63120731981\" (UID: \"9dab9b71-f170-46a5-b548-f63120731981\") " Jul 13 22:28:25.099892 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:25.099741 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dab9b71-f170-46a5-b548-f63120731981-istio-data\") pod \"9dab9b71-f170-46a5-b548-f63120731981\" (UID: \"9dab9b71-f170-46a5-b548-f63120731981\") " Jul 13 22:28:25.099892 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:25.099780 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dab9b71-f170-46a5-b548-f63120731981-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9dab9b71-f170-46a5-b548-f63120731981" (UID: "9dab9b71-f170-46a5-b548-f63120731981"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:28:25.100121 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:25.099964 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dab9b71-f170-46a5-b548-f63120731981-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9dab9b71-f170-46a5-b548-f63120731981" (UID: "9dab9b71-f170-46a5-b548-f63120731981"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:28:25.100121 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:25.099988 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dab9b71-f170-46a5-b548-f63120731981-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9dab9b71-f170-46a5-b548-f63120731981" (UID: "9dab9b71-f170-46a5-b548-f63120731981"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:28:25.100121 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:25.100045 2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dab9b71-f170-46a5-b548-f63120731981-credential-socket\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:28:25.100694 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:25.100662 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dab9b71-f170-46a5-b548-f63120731981-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9dab9b71-f170-46a5-b548-f63120731981" (UID: "9dab9b71-f170-46a5-b548-f63120731981"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:28:25.100843 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:25.100817 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dab9b71-f170-46a5-b548-f63120731981-istio-data" (OuterVolumeSpecName: "istio-data") pod "9dab9b71-f170-46a5-b548-f63120731981" (UID: "9dab9b71-f170-46a5-b548-f63120731981"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:28:25.101925 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:25.101904 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9dab9b71-f170-46a5-b548-f63120731981-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9dab9b71-f170-46a5-b548-f63120731981" (UID: "9dab9b71-f170-46a5-b548-f63120731981"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 13 22:28:25.102017 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:25.101900 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dab9b71-f170-46a5-b548-f63120731981-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9dab9b71-f170-46a5-b548-f63120731981" (UID: "9dab9b71-f170-46a5-b548-f63120731981"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:28:25.102017 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:25.101961 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dab9b71-f170-46a5-b548-f63120731981-istio-token" (OuterVolumeSpecName: "istio-token") pod "9dab9b71-f170-46a5-b548-f63120731981" (UID: "9dab9b71-f170-46a5-b548-f63120731981"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:28:25.102017 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:25.101961 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dab9b71-f170-46a5-b548-f63120731981-kube-api-access-jwcv8" (OuterVolumeSpecName: "kube-api-access-jwcv8") pod "9dab9b71-f170-46a5-b548-f63120731981" (UID: "9dab9b71-f170-46a5-b548-f63120731981"). InnerVolumeSpecName "kube-api-access-jwcv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:28:25.200590 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:25.200550 2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dab9b71-f170-46a5-b548-f63120731981-istio-podinfo\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:28:25.200590 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:25.200585 2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dab9b71-f170-46a5-b548-f63120731981-istio-envoy\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:28:25.200814 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:25.200599 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwcv8\" (UniqueName: \"kubernetes.io/projected/9dab9b71-f170-46a5-b548-f63120731981-kube-api-access-jwcv8\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:28:25.200814 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:25.200613 2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dab9b71-f170-46a5-b548-f63120731981-workload-certs\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:28:25.200814 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:25.200625 2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dab9b71-f170-46a5-b548-f63120731981-istio-token\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:28:25.200814 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:25.200636 2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dab9b71-f170-46a5-b548-f63120731981-istiod-ca-cert\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:28:25.200814 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:25.200646 2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dab9b71-f170-46a5-b548-f63120731981-workload-socket\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:28:25.200814 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:25.200657 2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dab9b71-f170-46a5-b548-f63120731981-istio-data\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:28:25.471524 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:25.471488 2555 generic.go:356] "Generic (PLEG): container finished" podID="9dab9b71-f170-46a5-b548-f63120731981" containerID="98c1ac726d7f59d89c33ef4b916e1810d7ddae60316bad6e0c6e128fb429c43e" exitCode=0 Jul 13 22:28:25.471713 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:25.471549 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s" Jul 13 22:28:25.471713 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:25.471577 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s" event={"ID":"9dab9b71-f170-46a5-b548-f63120731981","Type":"ContainerDied","Data":"98c1ac726d7f59d89c33ef4b916e1810d7ddae60316bad6e0c6e128fb429c43e"} Jul 13 22:28:25.471713 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:25.471641 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s" event={"ID":"9dab9b71-f170-46a5-b548-f63120731981","Type":"ContainerDied","Data":"39b5d119c65fbf3269202e0b499627cf946b17acbe72abf84dacd23f49a33cb2"} Jul 13 22:28:25.471713 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:25.471676 2555 scope.go:117] "RemoveContainer" containerID="98c1ac726d7f59d89c33ef4b916e1810d7ddae60316bad6e0c6e128fb429c43e" Jul 13 22:28:25.481438 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:25.481415 2555 scope.go:117] "RemoveContainer" containerID="9176f2f1493fe56d1f1ffde68979ff5f7c84944b090d943d84924d38b766b7cb" Jul 13 22:28:25.489991 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:25.489971 2555 scope.go:117] "RemoveContainer" containerID="98c1ac726d7f59d89c33ef4b916e1810d7ddae60316bad6e0c6e128fb429c43e" Jul 13 22:28:25.490287 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:28:25.490266 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98c1ac726d7f59d89c33ef4b916e1810d7ddae60316bad6e0c6e128fb429c43e\": container with ID starting with 98c1ac726d7f59d89c33ef4b916e1810d7ddae60316bad6e0c6e128fb429c43e not found: ID does not exist" containerID="98c1ac726d7f59d89c33ef4b916e1810d7ddae60316bad6e0c6e128fb429c43e" Jul 13 22:28:25.490367 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:25.490300 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98c1ac726d7f59d89c33ef4b916e1810d7ddae60316bad6e0c6e128fb429c43e"} err="failed to get container status \"98c1ac726d7f59d89c33ef4b916e1810d7ddae60316bad6e0c6e128fb429c43e\": rpc error: code = NotFound desc = could not find container \"98c1ac726d7f59d89c33ef4b916e1810d7ddae60316bad6e0c6e128fb429c43e\": container with ID starting with 98c1ac726d7f59d89c33ef4b916e1810d7ddae60316bad6e0c6e128fb429c43e not found: ID does not exist" Jul 13 22:28:25.490367 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:25.490327 2555 scope.go:117] "RemoveContainer" containerID="9176f2f1493fe56d1f1ffde68979ff5f7c84944b090d943d84924d38b766b7cb" Jul 13 22:28:25.490572 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:28:25.490551 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9176f2f1493fe56d1f1ffde68979ff5f7c84944b090d943d84924d38b766b7cb\": container with ID starting with 9176f2f1493fe56d1f1ffde68979ff5f7c84944b090d943d84924d38b766b7cb not found: ID does not exist" containerID="9176f2f1493fe56d1f1ffde68979ff5f7c84944b090d943d84924d38b766b7cb" Jul 13 22:28:25.490617 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:25.490580 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9176f2f1493fe56d1f1ffde68979ff5f7c84944b090d943d84924d38b766b7cb"} err="failed to get container status \"9176f2f1493fe56d1f1ffde68979ff5f7c84944b090d943d84924d38b766b7cb\": rpc error: code = NotFound desc = could not find container \"9176f2f1493fe56d1f1ffde68979ff5f7c84944b090d943d84924d38b766b7cb\": container with ID starting with 9176f2f1493fe56d1f1ffde68979ff5f7c84944b090d943d84924d38b766b7cb not found: ID does not exist" Jul 13 22:28:25.506331 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:25.506304 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s"] Jul 13 22:28:25.510884 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:25.510863 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-a399a528-openshift-default-77bcf9f859-vv55s"] Jul 13 22:28:25.612330 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:25.612298 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0081b233-f2l6j"] Jul 13 22:28:25.612661 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:25.612649 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dab9b71-f170-46a5-b548-f63120731981" containerName="istio-proxy" Jul 13 22:28:25.612661 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:25.612662 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dab9b71-f170-46a5-b548-f63120731981" containerName="istio-proxy" Jul 13 22:28:25.612748 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:25.612675 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dab9b71-f170-46a5-b548-f63120731981" containerName="istio-proxy" Jul 13 22:28:25.612748 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:25.612681 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dab9b71-f170-46a5-b548-f63120731981" containerName="istio-proxy" Jul 13 22:28:25.612748 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:25.612746 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="9dab9b71-f170-46a5-b548-f63120731981" containerName="istio-proxy" Jul 13 22:28:25.615577 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:25.615561 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0081b233-f2l6j" Jul 13 22:28:25.618486 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:25.618466 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k5z5d\"" Jul 13 22:28:25.618678 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:25.618666 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 13 22:28:25.633801 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:25.633771 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0081b233-f2l6j"] Jul 13 22:28:25.705381 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:25.705345 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6c29078-a608-4bf6-aef7-547c7424e9bd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0081b233-f2l6j\" (UID: \"e6c29078-a608-4bf6-aef7-547c7424e9bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0081b233-f2l6j" Jul 13 22:28:25.806670 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:25.806574 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6c29078-a608-4bf6-aef7-547c7424e9bd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0081b233-f2l6j\" (UID: \"e6c29078-a608-4bf6-aef7-547c7424e9bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0081b233-f2l6j" Jul 13 22:28:25.807256 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:25.807238 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6c29078-a608-4bf6-aef7-547c7424e9bd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0081b233-f2l6j\" (UID: \"e6c29078-a608-4bf6-aef7-547c7424e9bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0081b233-f2l6j" Jul 13 22:28:25.925975 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:25.925939 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0081b233-f2l6j" Jul 13 22:28:26.068488 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:26.068458 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0081b233-f2l6j"] Jul 13 22:28:26.069748 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:28:26.069720 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6c29078_a608_4bf6_aef7_547c7424e9bd.slice/crio-aa0c8b0b4371eade4c04b9c38cafa30b2f123d318bce8dc883a54d3db7b5bbb5 WatchSource:0}: Error finding container aa0c8b0b4371eade4c04b9c38cafa30b2f123d318bce8dc883a54d3db7b5bbb5: Status 404 returned error can't find the container with id aa0c8b0b4371eade4c04b9c38cafa30b2f123d318bce8dc883a54d3db7b5bbb5 Jul 13 22:28:26.478874 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:26.478838 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0081b233-f2l6j" event={"ID":"e6c29078-a608-4bf6-aef7-547c7424e9bd","Type":"ContainerStarted","Data":"739b3d4a77dd4718546786807013fff443369ed9071bc5de4b56ff3823e75332"} Jul 13 22:28:26.479037 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:26.478896 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0081b233-f2l6j" event={"ID":"e6c29078-a608-4bf6-aef7-547c7424e9bd","Type":"ContainerStarted","Data":"aa0c8b0b4371eade4c04b9c38cafa30b2f123d318bce8dc883a54d3db7b5bbb5"} Jul 13 22:28:26.497726 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:26.497680 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0081b233-f2l6j" podStartSLOduration=1.497663456 podStartE2EDuration="1.497663456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:28:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:28:26.49627278 +0000 UTC m=+1712.490027093" watchObservedRunningTime="2026-07-13 22:28:26.497663456 +0000 UTC m=+1712.491417792" Jul 13 22:28:26.560675 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:26.560639 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9dab9b71-f170-46a5-b548-f63120731981" path="/var/lib/kubelet/pods/9dab9b71-f170-46a5-b548-f63120731981/volumes" Jul 13 22:28:27.487288 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:27.487252 2555 generic.go:356] "Generic (PLEG): container finished" podID="e6c29078-a608-4bf6-aef7-547c7424e9bd" containerID="739b3d4a77dd4718546786807013fff443369ed9071bc5de4b56ff3823e75332" exitCode=7 Jul 13 22:28:27.487649 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:27.487344 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0081b233-f2l6j" event={"ID":"e6c29078-a608-4bf6-aef7-547c7424e9bd","Type":"ContainerDied","Data":"739b3d4a77dd4718546786807013fff443369ed9071bc5de4b56ff3823e75332"} Jul 13 22:28:27.487649 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:27.487563 2555 scope.go:117] "RemoveContainer" containerID="739b3d4a77dd4718546786807013fff443369ed9071bc5de4b56ff3823e75332" Jul 13 22:28:28.496356 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:28.496319 2555 generic.go:356] "Generic (PLEG): container finished" podID="e6c29078-a608-4bf6-aef7-547c7424e9bd" containerID="8bcae361d4f8d17676c77b18f7b90c3e5b9da400827c45b7b1079a36f65ef13e" exitCode=7 Jul 13 22:28:28.496791 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:28.496413 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0081b233-f2l6j" event={"ID":"e6c29078-a608-4bf6-aef7-547c7424e9bd","Type":"ContainerDied","Data":"8bcae361d4f8d17676c77b18f7b90c3e5b9da400827c45b7b1079a36f65ef13e"} Jul 13 22:28:28.496791 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:28.496537 2555 scope.go:117] "RemoveContainer" containerID="739b3d4a77dd4718546786807013fff443369ed9071bc5de4b56ff3823e75332" Jul 13 22:28:28.496791 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:28.496783 2555 scope.go:117] "RemoveContainer" containerID="8bcae361d4f8d17676c77b18f7b90c3e5b9da400827c45b7b1079a36f65ef13e" Jul 13 22:28:28.497034 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:28:28.497016 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-0081b233-f2l6j_odh-ai-gateway-infra(e6c29078-a608-4bf6-aef7-547c7424e9bd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0081b233-f2l6j" podUID="e6c29078-a608-4bf6-aef7-547c7424e9bd" Jul 13 22:28:29.503934 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:29.503900 2555 scope.go:117] "RemoveContainer" containerID="8bcae361d4f8d17676c77b18f7b90c3e5b9da400827c45b7b1079a36f65ef13e" Jul 13 22:28:29.504398 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:28:29.504145 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-0081b233-f2l6j_odh-ai-gateway-infra(e6c29078-a608-4bf6-aef7-547c7424e9bd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0081b233-f2l6j" podUID="e6c29078-a608-4bf6-aef7-547c7424e9bd" Jul 13 22:28:38.091415 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:38.090483 2555 patch_prober.go:28] interesting pod/e2e-derive-0081b233-openshift-default-5885f55d47-2wchr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": read tcp 10.134.0.2:37548->10.134.0.58:15021: read: connection reset by peer" start-of-body= Jul 13 22:28:38.091415 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:38.090569 2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-0081b233-openshift-default-5885f55d47-2wchr" podUID="4719b18b-0095-49a1-8712-e2a5616721cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": read tcp 10.134.0.2:37548->10.134.0.58:15021: read: connection reset by peer" Jul 13 22:28:38.563096 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:38.563066 2555 generic.go:356] "Generic (PLEG): container finished" podID="4719b18b-0095-49a1-8712-e2a5616721cb" containerID="4aa5d7f5534c4263e2aca046408b6f9f69e505ad82c6dd03e60798ee07a5cd65" exitCode=0 Jul 13 22:28:38.563289 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:38.563139 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0081b233-openshift-default-5885f55d47-2wchr" event={"ID":"4719b18b-0095-49a1-8712-e2a5616721cb","Type":"ContainerDied","Data":"4aa5d7f5534c4263e2aca046408b6f9f69e505ad82c6dd03e60798ee07a5cd65"} Jul 13 22:28:38.563550 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:38.563534 2555 scope.go:117] "RemoveContainer" containerID="4aa5d7f5534c4263e2aca046408b6f9f69e505ad82c6dd03e60798ee07a5cd65" Jul 13 22:28:38.564548 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:38.564519 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:28:38.564639 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:38.564604 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:28:38.564700 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:38.564656 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:28:39.570451 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:39.570415 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0081b233-openshift-default-5885f55d47-2wchr" event={"ID":"4719b18b-0095-49a1-8712-e2a5616721cb","Type":"ContainerStarted","Data":"f884ae72ef4c213a80f4fd483fca337b0ff808eb985c8c317c5282b3132cb7a0"} Jul 13 22:28:40.335076 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:40.335040 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-0081b233-openshift-default-5885f55d47-2wchr" Jul 13 22:28:40.335328 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:40.335304 2555 patch_prober.go:28] interesting pod/e2e-derive-0081b233-openshift-default-5885f55d47-2wchr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 13 22:28:40.335385 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:40.335365 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0081b233-openshift-default-5885f55d47-2wchr" podUID="4719b18b-0095-49a1-8712-e2a5616721cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 13 22:28:41.335134 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:41.335090 2555 patch_prober.go:28] interesting pod/e2e-derive-0081b233-openshift-default-5885f55d47-2wchr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 13 22:28:41.335547 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:41.335164 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0081b233-openshift-default-5885f55d47-2wchr" podUID="4719b18b-0095-49a1-8712-e2a5616721cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 13 22:28:42.334634 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:42.334602 2555 patch_prober.go:28] interesting pod/e2e-derive-0081b233-openshift-default-5885f55d47-2wchr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 13 22:28:42.334811 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:42.334664 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0081b233-openshift-default-5885f55d47-2wchr" podUID="4719b18b-0095-49a1-8712-e2a5616721cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 13 22:28:43.334580 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:43.334541 2555 patch_prober.go:28] interesting pod/e2e-derive-0081b233-openshift-default-5885f55d47-2wchr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 13 22:28:43.334968 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:43.334620 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0081b233-openshift-default-5885f55d47-2wchr" podUID="4719b18b-0095-49a1-8712-e2a5616721cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 13 22:28:43.553948 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:43.553919 2555 scope.go:117] "RemoveContainer" containerID="8bcae361d4f8d17676c77b18f7b90c3e5b9da400827c45b7b1079a36f65ef13e" Jul 13 22:28:44.337675 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:44.337643 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-0081b233-openshift-default-5885f55d47-2wchr" Jul 13 22:28:44.338073 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:44.338058 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-0081b233-openshift-default-5885f55d47-2wchr" Jul 13 22:28:44.338690 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:44.338672 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-0081b233-openshift-default-5885f55d47-2wchr" Jul 13 22:28:44.603414 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:44.603377 2555 generic.go:356] "Generic (PLEG): container finished" podID="e6c29078-a608-4bf6-aef7-547c7424e9bd" containerID="0dd71d1580bfaca9e675395c7b5a706ff00fe73594e1bbbdc90305960279414f" exitCode=0 Jul 13 22:28:44.603580 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:44.603443 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0081b233-f2l6j" event={"ID":"e6c29078-a608-4bf6-aef7-547c7424e9bd","Type":"ContainerDied","Data":"0dd71d1580bfaca9e675395c7b5a706ff00fe73594e1bbbdc90305960279414f"} Jul 13 22:28:44.603580 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:44.603496 2555 scope.go:117] "RemoveContainer" containerID="8bcae361d4f8d17676c77b18f7b90c3e5b9da400827c45b7b1079a36f65ef13e" Jul 13 22:28:45.633968 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:45.633931 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0081b233-f2l6j"] Jul 13 22:28:45.759514 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:45.759488 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0081b233-f2l6j" Jul 13 22:28:45.784268 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:45.784236 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6c29078-a608-4bf6-aef7-547c7424e9bd-maas-api-service-ca\") pod \"e6c29078-a608-4bf6-aef7-547c7424e9bd\" (UID: \"e6c29078-a608-4bf6-aef7-547c7424e9bd\") " Jul 13 22:28:45.784575 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:45.784553 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6c29078-a608-4bf6-aef7-547c7424e9bd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e6c29078-a608-4bf6-aef7-547c7424e9bd" (UID: "e6c29078-a608-4bf6-aef7-547c7424e9bd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:28:45.885259 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:45.885192 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6c29078-a608-4bf6-aef7-547c7424e9bd-maas-api-service-ca\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:28:46.616095 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:46.616066 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0081b233-f2l6j" Jul 13 22:28:46.616095 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:46.616073 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0081b233-f2l6j" event={"ID":"e6c29078-a608-4bf6-aef7-547c7424e9bd","Type":"ContainerDied","Data":"aa0c8b0b4371eade4c04b9c38cafa30b2f123d318bce8dc883a54d3db7b5bbb5"} Jul 13 22:28:46.616332 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:46.616126 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa0c8b0b4371eade4c04b9c38cafa30b2f123d318bce8dc883a54d3db7b5bbb5" Jul 13 22:28:46.647095 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:46.647068 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0081b233-f2l6j"] Jul 13 22:28:46.650745 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:46.650724 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0081b233-f2l6j"] Jul 13 22:28:48.559729 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:28:48.559693 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6c29078-a608-4bf6-aef7-547c7424e9bd" path="/var/lib/kubelet/pods/e6c29078-a608-4bf6-aef7-547c7424e9bd/volumes" Jul 13 22:29:25.668308 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:25.668274 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0081b233-mtb4m"] Jul 13 22:29:25.668775 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:25.668648 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6c29078-a608-4bf6-aef7-547c7424e9bd" containerName="revoke-keys" Jul 13 22:29:25.668775 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:25.668659 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6c29078-a608-4bf6-aef7-547c7424e9bd" containerName="revoke-keys" Jul 13 22:29:25.668775 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:25.668684 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6c29078-a608-4bf6-aef7-547c7424e9bd" containerName="revoke-keys" Jul 13 22:29:25.668775 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:25.668689 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6c29078-a608-4bf6-aef7-547c7424e9bd" containerName="revoke-keys" Jul 13 22:29:25.668775 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:25.668740 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6c29078-a608-4bf6-aef7-547c7424e9bd" containerName="revoke-keys" Jul 13 22:29:25.668775 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:25.668750 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6c29078-a608-4bf6-aef7-547c7424e9bd" containerName="revoke-keys" Jul 13 22:29:25.668775 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:25.668758 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6c29078-a608-4bf6-aef7-547c7424e9bd" containerName="revoke-keys" Jul 13 22:29:25.668775 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:25.668765 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="9dab9b71-f170-46a5-b548-f63120731981" containerName="istio-proxy" Jul 13 22:29:25.671722 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:25.671706 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0081b233-mtb4m" Jul 13 22:29:25.674320 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:25.674298 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 13 22:29:25.674426 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:25.674304 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k5z5d\"" Jul 13 22:29:25.680302 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:25.680280 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0081b233-mtb4m"] Jul 13 22:29:25.736182 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:25.736153 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53051199-f7d4-4517-baf7-603ab478340f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0081b233-mtb4m\" (UID: \"53051199-f7d4-4517-baf7-603ab478340f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0081b233-mtb4m" Jul 13 22:29:25.836932 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:25.836898 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53051199-f7d4-4517-baf7-603ab478340f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0081b233-mtb4m\" (UID: \"53051199-f7d4-4517-baf7-603ab478340f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0081b233-mtb4m" Jul 13 22:29:25.837504 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:25.837483 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53051199-f7d4-4517-baf7-603ab478340f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0081b233-mtb4m\" (UID: \"53051199-f7d4-4517-baf7-603ab478340f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0081b233-mtb4m" Jul 13 22:29:25.983455 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:25.983378 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0081b233-mtb4m" Jul 13 22:29:26.117664 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:26.117639 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0081b233-mtb4m"] Jul 13 22:29:26.119280 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:29:26.119255 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53051199_f7d4_4517_baf7_603ab478340f.slice/crio-be92cbc3f82b76ef8f9b0e0434c0902018d6efd6830e502cff23b269a4449244 WatchSource:0}: Error finding container be92cbc3f82b76ef8f9b0e0434c0902018d6efd6830e502cff23b269a4449244: Status 404 returned error can't find the container with id be92cbc3f82b76ef8f9b0e0434c0902018d6efd6830e502cff23b269a4449244 Jul 13 22:29:26.849609 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:26.849578 2555 generic.go:356] "Generic (PLEG): container finished" podID="53051199-f7d4-4517-baf7-603ab478340f" containerID="a410b9878400ce66950d8e0f9740efd0cb45c540813e3e3b4dbc6fa06a4fb921" exitCode=0 Jul 13 22:29:26.850006 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:26.849663 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0081b233-mtb4m" event={"ID":"53051199-f7d4-4517-baf7-603ab478340f","Type":"ContainerDied","Data":"a410b9878400ce66950d8e0f9740efd0cb45c540813e3e3b4dbc6fa06a4fb921"} Jul 13 22:29:26.850006 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:26.849732 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0081b233-mtb4m" event={"ID":"53051199-f7d4-4517-baf7-603ab478340f","Type":"ContainerStarted","Data":"be92cbc3f82b76ef8f9b0e0434c0902018d6efd6830e502cff23b269a4449244"} Jul 13 22:29:27.993019 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:27.992995 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0081b233-mtb4m" Jul 13 22:29:28.055311 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:28.055282 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53051199-f7d4-4517-baf7-603ab478340f-maas-api-service-ca\") pod \"53051199-f7d4-4517-baf7-603ab478340f\" (UID: \"53051199-f7d4-4517-baf7-603ab478340f\") " Jul 13 22:29:28.055682 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:28.055658 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53051199-f7d4-4517-baf7-603ab478340f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "53051199-f7d4-4517-baf7-603ab478340f" (UID: "53051199-f7d4-4517-baf7-603ab478340f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:29:28.156337 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:28.156309 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53051199-f7d4-4517-baf7-603ab478340f-maas-api-service-ca\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:29:28.861338 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:28.861253 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0081b233-mtb4m" Jul 13 22:29:28.861483 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:28.861262 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0081b233-mtb4m" event={"ID":"53051199-f7d4-4517-baf7-603ab478340f","Type":"ContainerDied","Data":"be92cbc3f82b76ef8f9b0e0434c0902018d6efd6830e502cff23b269a4449244"} Jul 13 22:29:28.861483 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:28.861397 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be92cbc3f82b76ef8f9b0e0434c0902018d6efd6830e502cff23b269a4449244" Jul 13 22:29:50.934252 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:50.934161 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-0081b233-openshift-default-5885f55d47-2wchr"] Jul 13 22:29:50.934649 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:50.934405 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-0081b233-openshift-default-5885f55d47-2wchr" podUID="4719b18b-0095-49a1-8712-e2a5616721cb" containerName="istio-proxy" containerID="cri-o://f884ae72ef4c213a80f4fd483fca337b0ff808eb985c8c317c5282b3132cb7a0" gracePeriod=30 Jul 13 22:29:56.198121 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:56.198050 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0081b233-openshift-default-5885f55d47-2wchr" Jul 13 22:29:56.309602 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:56.309572 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9hdbw\" (UniqueName: \"kubernetes.io/projected/4719b18b-0095-49a1-8712-e2a5616721cb-kube-api-access-9hdbw\") pod \"4719b18b-0095-49a1-8712-e2a5616721cb\" (UID: \"4719b18b-0095-49a1-8712-e2a5616721cb\") " Jul 13 22:29:56.309758 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:56.309617 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4719b18b-0095-49a1-8712-e2a5616721cb-istio-data\") pod \"4719b18b-0095-49a1-8712-e2a5616721cb\" (UID: \"4719b18b-0095-49a1-8712-e2a5616721cb\") " Jul 13 22:29:56.309758 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:56.309639 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4719b18b-0095-49a1-8712-e2a5616721cb-istio-podinfo\") pod \"4719b18b-0095-49a1-8712-e2a5616721cb\" (UID: \"4719b18b-0095-49a1-8712-e2a5616721cb\") " Jul 13 22:29:56.309758 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:56.309664 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4719b18b-0095-49a1-8712-e2a5616721cb-workload-certs\") pod \"4719b18b-0095-49a1-8712-e2a5616721cb\" (UID: \"4719b18b-0095-49a1-8712-e2a5616721cb\") " Jul 13 22:29:56.309930 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:56.309780 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4719b18b-0095-49a1-8712-e2a5616721cb-istio-envoy\") pod \"4719b18b-0095-49a1-8712-e2a5616721cb\" (UID: \"4719b18b-0095-49a1-8712-e2a5616721cb\") " Jul 13 22:29:56.309930 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:56.309837 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4719b18b-0095-49a1-8712-e2a5616721cb-workload-socket\") pod \"4719b18b-0095-49a1-8712-e2a5616721cb\" (UID: \"4719b18b-0095-49a1-8712-e2a5616721cb\") " Jul 13 22:29:56.309930 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:56.309866 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4719b18b-0095-49a1-8712-e2a5616721cb-istiod-ca-cert\") pod \"4719b18b-0095-49a1-8712-e2a5616721cb\" (UID: \"4719b18b-0095-49a1-8712-e2a5616721cb\") " Jul 13 22:29:56.310098 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:56.309936 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4719b18b-0095-49a1-8712-e2a5616721cb-istio-token\") pod \"4719b18b-0095-49a1-8712-e2a5616721cb\" (UID: \"4719b18b-0095-49a1-8712-e2a5616721cb\") " Jul 13 22:29:56.310098 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:56.309963 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4719b18b-0095-49a1-8712-e2a5616721cb-credential-socket\") pod \"4719b18b-0095-49a1-8712-e2a5616721cb\" (UID: \"4719b18b-0095-49a1-8712-e2a5616721cb\") " Jul 13 22:29:56.310098 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:56.309980 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4719b18b-0095-49a1-8712-e2a5616721cb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4719b18b-0095-49a1-8712-e2a5616721cb" (UID: "4719b18b-0095-49a1-8712-e2a5616721cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:29:56.310301 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:56.310267 2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4719b18b-0095-49a1-8712-e2a5616721cb-workload-certs\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:29:56.310422 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:56.310322 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4719b18b-0095-49a1-8712-e2a5616721cb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4719b18b-0095-49a1-8712-e2a5616721cb" (UID: "4719b18b-0095-49a1-8712-e2a5616721cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:29:56.310422 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:56.310385 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4719b18b-0095-49a1-8712-e2a5616721cb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4719b18b-0095-49a1-8712-e2a5616721cb" (UID: "4719b18b-0095-49a1-8712-e2a5616721cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:29:56.310669 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:56.310645 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4719b18b-0095-49a1-8712-e2a5616721cb-istio-data" (OuterVolumeSpecName: "istio-data") pod "4719b18b-0095-49a1-8712-e2a5616721cb" (UID: "4719b18b-0095-49a1-8712-e2a5616721cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:29:56.310669 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:56.310660 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4719b18b-0095-49a1-8712-e2a5616721cb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4719b18b-0095-49a1-8712-e2a5616721cb" (UID: "4719b18b-0095-49a1-8712-e2a5616721cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:29:56.311968 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:56.311946 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4719b18b-0095-49a1-8712-e2a5616721cb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4719b18b-0095-49a1-8712-e2a5616721cb" (UID: "4719b18b-0095-49a1-8712-e2a5616721cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 13 22:29:56.312069 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:56.311972 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4719b18b-0095-49a1-8712-e2a5616721cb-kube-api-access-9hdbw" (OuterVolumeSpecName: "kube-api-access-9hdbw") pod "4719b18b-0095-49a1-8712-e2a5616721cb" (UID: "4719b18b-0095-49a1-8712-e2a5616721cb"). InnerVolumeSpecName "kube-api-access-9hdbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:29:56.312495 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:56.312477 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4719b18b-0095-49a1-8712-e2a5616721cb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4719b18b-0095-49a1-8712-e2a5616721cb" (UID: "4719b18b-0095-49a1-8712-e2a5616721cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:29:56.312495 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:56.312479 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4719b18b-0095-49a1-8712-e2a5616721cb-istio-token" (OuterVolumeSpecName: "istio-token") pod "4719b18b-0095-49a1-8712-e2a5616721cb" (UID: "4719b18b-0095-49a1-8712-e2a5616721cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:29:56.411427 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:56.411400 2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4719b18b-0095-49a1-8712-e2a5616721cb-istiod-ca-cert\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:29:56.411427 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:56.411425 2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4719b18b-0095-49a1-8712-e2a5616721cb-istio-token\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:29:56.411600 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:56.411435 2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4719b18b-0095-49a1-8712-e2a5616721cb-credential-socket\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:29:56.411600 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:56.411444 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hdbw\" (UniqueName: \"kubernetes.io/projected/4719b18b-0095-49a1-8712-e2a5616721cb-kube-api-access-9hdbw\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:29:56.411600 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:56.411454 2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4719b18b-0095-49a1-8712-e2a5616721cb-istio-data\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:29:56.411600 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:56.411463 2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4719b18b-0095-49a1-8712-e2a5616721cb-istio-podinfo\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:29:56.411600 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:56.411472 2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4719b18b-0095-49a1-8712-e2a5616721cb-istio-envoy\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:29:56.411600 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:56.411480 2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4719b18b-0095-49a1-8712-e2a5616721cb-workload-socket\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:29:57.032222 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:57.032167 2555 generic.go:356] "Generic (PLEG): container finished" podID="4719b18b-0095-49a1-8712-e2a5616721cb" containerID="f884ae72ef4c213a80f4fd483fca337b0ff808eb985c8c317c5282b3132cb7a0" exitCode=0 Jul 13 22:29:57.032477 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:57.032256 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0081b233-openshift-default-5885f55d47-2wchr" Jul 13 22:29:57.032477 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:57.032255 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0081b233-openshift-default-5885f55d47-2wchr" event={"ID":"4719b18b-0095-49a1-8712-e2a5616721cb","Type":"ContainerDied","Data":"f884ae72ef4c213a80f4fd483fca337b0ff808eb985c8c317c5282b3132cb7a0"} Jul 13 22:29:57.032477 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:57.032350 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0081b233-openshift-default-5885f55d47-2wchr" event={"ID":"4719b18b-0095-49a1-8712-e2a5616721cb","Type":"ContainerDied","Data":"d1bcfe1b0e5732e53b71f0234badd44dda7e7801746f41e7101e3439ed25e011"} Jul 13 22:29:57.032477 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:57.032383 2555 scope.go:117] "RemoveContainer" containerID="f884ae72ef4c213a80f4fd483fca337b0ff808eb985c8c317c5282b3132cb7a0" Jul 13 22:29:57.042054 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:57.042037 2555 scope.go:117] "RemoveContainer" containerID="4aa5d7f5534c4263e2aca046408b6f9f69e505ad82c6dd03e60798ee07a5cd65" Jul 13 22:29:57.050991 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:57.050968 2555 scope.go:117] "RemoveContainer" containerID="f884ae72ef4c213a80f4fd483fca337b0ff808eb985c8c317c5282b3132cb7a0" Jul 13 22:29:57.051333 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:29:57.051311 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f884ae72ef4c213a80f4fd483fca337b0ff808eb985c8c317c5282b3132cb7a0\": container with ID starting with f884ae72ef4c213a80f4fd483fca337b0ff808eb985c8c317c5282b3132cb7a0 not found: ID does not exist" containerID="f884ae72ef4c213a80f4fd483fca337b0ff808eb985c8c317c5282b3132cb7a0" Jul 13 22:29:57.051430 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:57.051341 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f884ae72ef4c213a80f4fd483fca337b0ff808eb985c8c317c5282b3132cb7a0"} err="failed to get container status \"f884ae72ef4c213a80f4fd483fca337b0ff808eb985c8c317c5282b3132cb7a0\": rpc error: code = NotFound desc = could not find container \"f884ae72ef4c213a80f4fd483fca337b0ff808eb985c8c317c5282b3132cb7a0\": container with ID starting with f884ae72ef4c213a80f4fd483fca337b0ff808eb985c8c317c5282b3132cb7a0 not found: ID does not exist" Jul 13 22:29:57.051430 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:57.051358 2555 scope.go:117] "RemoveContainer" containerID="4aa5d7f5534c4263e2aca046408b6f9f69e505ad82c6dd03e60798ee07a5cd65" Jul 13 22:29:57.051657 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:29:57.051637 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4aa5d7f5534c4263e2aca046408b6f9f69e505ad82c6dd03e60798ee07a5cd65\": container with ID starting with 4aa5d7f5534c4263e2aca046408b6f9f69e505ad82c6dd03e60798ee07a5cd65 not found: ID does not exist" containerID="4aa5d7f5534c4263e2aca046408b6f9f69e505ad82c6dd03e60798ee07a5cd65" Jul 13 22:29:57.051720 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:57.051666 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4aa5d7f5534c4263e2aca046408b6f9f69e505ad82c6dd03e60798ee07a5cd65"} err="failed to get container status \"4aa5d7f5534c4263e2aca046408b6f9f69e505ad82c6dd03e60798ee07a5cd65\": rpc error: code = NotFound desc = could not find container \"4aa5d7f5534c4263e2aca046408b6f9f69e505ad82c6dd03e60798ee07a5cd65\": container with ID starting with 4aa5d7f5534c4263e2aca046408b6f9f69e505ad82c6dd03e60798ee07a5cd65 not found: ID does not exist" Jul 13 22:29:57.054296 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:57.054271 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-0081b233-openshift-default-5885f55d47-2wchr"] Jul 13 22:29:57.057077 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:57.057057 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-0081b233-openshift-default-5885f55d47-2wchr"] Jul 13 22:29:58.559624 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:29:58.559590 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4719b18b-0095-49a1-8712-e2a5616721cb" path="/var/lib/kubelet/pods/4719b18b-0095-49a1-8712-e2a5616721cb/volumes" Jul 13 22:30:00.142357 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:30:00.142323 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29733030-mqz8r"] Jul 13 22:30:00.142744 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:30:00.142681 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4719b18b-0095-49a1-8712-e2a5616721cb" containerName="istio-proxy" Jul 13 22:30:00.142744 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:30:00.142693 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4719b18b-0095-49a1-8712-e2a5616721cb" containerName="istio-proxy" Jul 13 22:30:00.142744 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:30:00.142709 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53051199-f7d4-4517-baf7-603ab478340f" containerName="revoke-keys" Jul 13 22:30:00.142744 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:30:00.142714 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="53051199-f7d4-4517-baf7-603ab478340f" containerName="revoke-keys" Jul 13 22:30:00.142744 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:30:00.142723 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6c29078-a608-4bf6-aef7-547c7424e9bd" containerName="revoke-keys" Jul 13 22:30:00.142744 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:30:00.142728 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6c29078-a608-4bf6-aef7-547c7424e9bd" containerName="revoke-keys" Jul 13 22:30:00.142940 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:30:00.142775 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4719b18b-0095-49a1-8712-e2a5616721cb" containerName="istio-proxy" Jul 13 22:30:00.142940 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:30:00.142786 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="53051199-f7d4-4517-baf7-603ab478340f" containerName="revoke-keys" Jul 13 22:30:00.147374 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:30:00.147355 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733030-mqz8r" Jul 13 22:30:00.149938 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:30:00.149912 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k5z5d\"" Jul 13 22:30:00.155584 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:30:00.155563 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29733030-mqz8r"] Jul 13 22:30:00.159033 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:30:00.159012 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733030-mqz8r" Jul 13 22:30:00.299686 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:30:00.299662 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29733030-mqz8r"] Jul 13 22:30:00.302074 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:30:00.302049 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a20a0ca_1d06_4159_b5fd_e61a44112d6d.slice/crio-5957565d830e7ca607960fdf35f5d0baa775bc0891235acd6c645cee044198ef WatchSource:0}: Error finding container 5957565d830e7ca607960fdf35f5d0baa775bc0891235acd6c645cee044198ef: Status 404 returned error can't find the container with id 5957565d830e7ca607960fdf35f5d0baa775bc0891235acd6c645cee044198ef Jul 13 22:30:01.059452 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:30:01.059416 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733030-mqz8r" event={"ID":"5a20a0ca-1d06-4159-b5fd-e61a44112d6d","Type":"ContainerStarted","Data":"c70693c7ce8fc2708d8cb292f32ff375acdef301cd053f23670538668cb59f34"} Jul 13 22:30:01.059671 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:30:01.059479 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733030-mqz8r" event={"ID":"5a20a0ca-1d06-4159-b5fd-e61a44112d6d","Type":"ContainerStarted","Data":"5957565d830e7ca607960fdf35f5d0baa775bc0891235acd6c645cee044198ef"} Jul 13 22:30:01.076612 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:30:01.076561 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733030-mqz8r" podStartSLOduration=1.076543917 podStartE2EDuration="1.076543917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:30:01.074781791 +0000 UTC m=+1807.068536099" watchObservedRunningTime="2026-07-13 22:30:01.076543917 +0000 UTC m=+1807.070298210" Jul 13 22:30:02.066020 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:30:02.065982 2555 generic.go:356] "Generic (PLEG): container finished" podID="5a20a0ca-1d06-4159-b5fd-e61a44112d6d" containerID="c70693c7ce8fc2708d8cb292f32ff375acdef301cd053f23670538668cb59f34" exitCode=0 Jul 13 22:30:02.066436 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:30:02.066076 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733030-mqz8r" event={"ID":"5a20a0ca-1d06-4159-b5fd-e61a44112d6d","Type":"ContainerDied","Data":"c70693c7ce8fc2708d8cb292f32ff375acdef301cd053f23670538668cb59f34"} Jul 13 22:30:03.201026 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:30:03.200999 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733030-mqz8r" Jul 13 22:30:04.079030 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:30:04.078996 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733030-mqz8r" event={"ID":"5a20a0ca-1d06-4159-b5fd-e61a44112d6d","Type":"ContainerDied","Data":"5957565d830e7ca607960fdf35f5d0baa775bc0891235acd6c645cee044198ef"} Jul 13 22:30:04.079191 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:30:04.079045 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5957565d830e7ca607960fdf35f5d0baa775bc0891235acd6c645cee044198ef" Jul 13 22:30:04.079191 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:30:04.079010 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733030-mqz8r" Jul 13 22:30:54.912008 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:30:54.911975 2555 scope.go:117] "RemoveContainer" containerID="97ff9cc82b146733cc928a5e3269b9b97d6e95cbb29e3ac360f1d4b8c6b82e6b" Jul 13 22:31:27.256966 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:27.256874 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk"] Jul 13 22:31:27.257563 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:27.257350 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a20a0ca-1d06-4159-b5fd-e61a44112d6d" containerName="cleanup" Jul 13 22:31:27.257563 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:27.257370 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a20a0ca-1d06-4159-b5fd-e61a44112d6d" containerName="cleanup" Jul 13 22:31:27.257563 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:27.257396 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4719b18b-0095-49a1-8712-e2a5616721cb" containerName="istio-proxy" Jul 13 22:31:27.257563 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:27.257405 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4719b18b-0095-49a1-8712-e2a5616721cb" containerName="istio-proxy" Jul 13 22:31:27.257563 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:27.257488 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4719b18b-0095-49a1-8712-e2a5616721cb" containerName="istio-proxy" Jul 13 22:31:27.257563 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:27.257501 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a20a0ca-1d06-4159-b5fd-e61a44112d6d" containerName="cleanup" Jul 13 22:31:27.260775 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:27.260754 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk" Jul 13 22:31:27.263731 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:27.263712 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-2fa7da44-openshift-default-dockercfg-zbm79\"" Jul 13 22:31:27.276024 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:27.275994 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk"] Jul 13 22:31:27.407577 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:27.407539 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f94fa89-2178-4ff5-a738-58e4e622ce68-istio-podinfo\") pod \"e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk\" (UID: \"7f94fa89-2178-4ff5-a738-58e4e622ce68\") " pod="openshift-ingress/e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk" Jul 13 22:31:27.407767 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:27.407586 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f94fa89-2178-4ff5-a738-58e4e622ce68-istio-envoy\") pod \"e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk\" (UID: \"7f94fa89-2178-4ff5-a738-58e4e622ce68\") " pod="openshift-ingress/e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk" Jul 13 22:31:27.407767 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:27.407613 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f94fa89-2178-4ff5-a738-58e4e622ce68-istiod-ca-cert\") pod \"e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk\" (UID: \"7f94fa89-2178-4ff5-a738-58e4e622ce68\") " pod="openshift-ingress/e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk" Jul 13 22:31:27.407767 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:27.407725 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f94fa89-2178-4ff5-a738-58e4e622ce68-workload-socket\") pod \"e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk\" (UID: \"7f94fa89-2178-4ff5-a738-58e4e622ce68\") " pod="openshift-ingress/e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk" Jul 13 22:31:27.407882 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:27.407794 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f94fa89-2178-4ff5-a738-58e4e622ce68-workload-certs\") pod \"e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk\" (UID: \"7f94fa89-2178-4ff5-a738-58e4e622ce68\") " pod="openshift-ingress/e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk" Jul 13 22:31:27.407882 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:27.407838 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f94fa89-2178-4ff5-a738-58e4e622ce68-istio-token\") pod \"e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk\" (UID: \"7f94fa89-2178-4ff5-a738-58e4e622ce68\") " pod="openshift-ingress/e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk" Jul 13 22:31:27.407946 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:27.407907 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f94fa89-2178-4ff5-a738-58e4e622ce68-credential-socket\") pod \"e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk\" (UID: \"7f94fa89-2178-4ff5-a738-58e4e622ce68\") " pod="openshift-ingress/e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk" Jul 13 22:31:27.407985 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:27.407942 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq4cg\" (UniqueName: \"kubernetes.io/projected/7f94fa89-2178-4ff5-a738-58e4e622ce68-kube-api-access-lq4cg\") pod \"e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk\" (UID: \"7f94fa89-2178-4ff5-a738-58e4e622ce68\") " pod="openshift-ingress/e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk" Jul 13 22:31:27.407985 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:27.407975 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f94fa89-2178-4ff5-a738-58e4e622ce68-istio-data\") pod \"e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk\" (UID: \"7f94fa89-2178-4ff5-a738-58e4e622ce68\") " pod="openshift-ingress/e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk" Jul 13 22:31:27.509302 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:27.509224 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f94fa89-2178-4ff5-a738-58e4e622ce68-istio-podinfo\") pod \"e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk\" (UID: \"7f94fa89-2178-4ff5-a738-58e4e622ce68\") " pod="openshift-ingress/e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk" Jul 13 22:31:27.509302 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:27.509260 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f94fa89-2178-4ff5-a738-58e4e622ce68-istio-envoy\") pod \"e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk\" (UID: \"7f94fa89-2178-4ff5-a738-58e4e622ce68\") " pod="openshift-ingress/e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk" Jul 13 22:31:27.509302 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:27.509287 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f94fa89-2178-4ff5-a738-58e4e622ce68-istiod-ca-cert\") pod \"e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk\" (UID: \"7f94fa89-2178-4ff5-a738-58e4e622ce68\") " pod="openshift-ingress/e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk" Jul 13 22:31:27.509540 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:27.509342 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f94fa89-2178-4ff5-a738-58e4e622ce68-workload-socket\") pod \"e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk\" (UID: \"7f94fa89-2178-4ff5-a738-58e4e622ce68\") " pod="openshift-ingress/e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk" Jul 13 22:31:27.509540 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:27.509412 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f94fa89-2178-4ff5-a738-58e4e622ce68-workload-certs\") pod \"e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk\" (UID: \"7f94fa89-2178-4ff5-a738-58e4e622ce68\") " pod="openshift-ingress/e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk" Jul 13 22:31:27.509540 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:27.509435 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f94fa89-2178-4ff5-a738-58e4e622ce68-istio-token\") pod \"e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk\" (UID: \"7f94fa89-2178-4ff5-a738-58e4e622ce68\") " pod="openshift-ingress/e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk" Jul 13 22:31:27.509540 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:27.509473 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f94fa89-2178-4ff5-a738-58e4e622ce68-credential-socket\") pod \"e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk\" (UID: \"7f94fa89-2178-4ff5-a738-58e4e622ce68\") " pod="openshift-ingress/e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk" Jul 13 22:31:27.509540 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:27.509496 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq4cg\" (UniqueName: \"kubernetes.io/projected/7f94fa89-2178-4ff5-a738-58e4e622ce68-kube-api-access-lq4cg\") pod \"e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk\" (UID: \"7f94fa89-2178-4ff5-a738-58e4e622ce68\") " pod="openshift-ingress/e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk" Jul 13 22:31:27.509540 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:27.509526 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f94fa89-2178-4ff5-a738-58e4e622ce68-istio-data\") pod \"e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk\" (UID: \"7f94fa89-2178-4ff5-a738-58e4e622ce68\") " pod="openshift-ingress/e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk" Jul 13 22:31:27.509827 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:27.509773 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f94fa89-2178-4ff5-a738-58e4e622ce68-workload-certs\") pod \"e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk\" (UID: \"7f94fa89-2178-4ff5-a738-58e4e622ce68\") " pod="openshift-ingress/e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk" Jul 13 22:31:27.509827 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:27.509811 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f94fa89-2178-4ff5-a738-58e4e622ce68-workload-socket\") pod \"e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk\" (UID: \"7f94fa89-2178-4ff5-a738-58e4e622ce68\") " pod="openshift-ingress/e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk" Jul 13 22:31:27.510071 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:27.509968 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f94fa89-2178-4ff5-a738-58e4e622ce68-credential-socket\") pod \"e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk\" (UID: \"7f94fa89-2178-4ff5-a738-58e4e622ce68\") " pod="openshift-ingress/e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk" Jul 13 22:31:27.510412 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:27.510390 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f94fa89-2178-4ff5-a738-58e4e622ce68-istio-data\") pod \"e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk\" (UID: \"7f94fa89-2178-4ff5-a738-58e4e622ce68\") " pod="openshift-ingress/e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk" Jul 13 22:31:27.510487 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:27.510447 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f94fa89-2178-4ff5-a738-58e4e622ce68-istiod-ca-cert\") pod \"e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk\" (UID: \"7f94fa89-2178-4ff5-a738-58e4e622ce68\") " pod="openshift-ingress/e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk" Jul 13 22:31:27.511849 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:27.511822 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f94fa89-2178-4ff5-a738-58e4e622ce68-istio-envoy\") pod \"e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk\" (UID: \"7f94fa89-2178-4ff5-a738-58e4e622ce68\") " pod="openshift-ingress/e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk" Jul 13 22:31:27.512140 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:27.512113 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f94fa89-2178-4ff5-a738-58e4e622ce68-istio-podinfo\") pod \"e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk\" (UID: \"7f94fa89-2178-4ff5-a738-58e4e622ce68\") " pod="openshift-ingress/e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk" Jul 13 22:31:27.519274 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:27.519251 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f94fa89-2178-4ff5-a738-58e4e622ce68-istio-token\") pod \"e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk\" (UID: \"7f94fa89-2178-4ff5-a738-58e4e622ce68\") " pod="openshift-ingress/e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk" Jul 13 22:31:27.519402 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:27.519294 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq4cg\" (UniqueName: \"kubernetes.io/projected/7f94fa89-2178-4ff5-a738-58e4e622ce68-kube-api-access-lq4cg\") pod \"e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk\" (UID: \"7f94fa89-2178-4ff5-a738-58e4e622ce68\") " pod="openshift-ingress/e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk" Jul 13 22:31:27.571477 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:27.571438 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk" Jul 13 22:31:27.768380 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:27.768347 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk"] Jul 13 22:31:27.771088 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:31:27.771062 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f94fa89_2178_4ff5_a738_58e4e622ce68.slice/crio-e6bb9ca68d8445f30626c10ea99e6731f4ccb9c3ea1c93430468d7f209646e31 WatchSource:0}: Error finding container e6bb9ca68d8445f30626c10ea99e6731f4ccb9c3ea1c93430468d7f209646e31: Status 404 returned error can't find the container with id e6bb9ca68d8445f30626c10ea99e6731f4ccb9c3ea1c93430468d7f209646e31 Jul 13 22:31:27.773708 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:27.773677 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:31:27.773788 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:27.773750 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:31:27.773788 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:27.773776 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:31:28.551093 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:28.551054 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk" event={"ID":"7f94fa89-2178-4ff5-a738-58e4e622ce68","Type":"ContainerStarted","Data":"df385043694a7e88b47dcdeecb573498d1373dfd065aff07797a569a38de366e"} Jul 13 22:31:28.551600 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:28.551120 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk" event={"ID":"7f94fa89-2178-4ff5-a738-58e4e622ce68","Type":"ContainerStarted","Data":"e6bb9ca68d8445f30626c10ea99e6731f4ccb9c3ea1c93430468d7f209646e31"} Jul 13 22:31:28.572499 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:28.572468 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk" Jul 13 22:31:28.577961 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:28.577929 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk" Jul 13 22:31:28.578401 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:28.578359 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk" podStartSLOduration=1.57834344 podStartE2EDuration="1.57834344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:31:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:31:28.575193917 +0000 UTC m=+1894.568948230" watchObservedRunningTime="2026-07-13 22:31:28.57834344 +0000 UTC m=+1894.572097795" Jul 13 22:31:29.558006 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:29.557976 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk" Jul 13 22:31:29.559247 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:29.559229 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk" Jul 13 22:31:42.278588 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:42.278547 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2fa7da44-h2x7p"] Jul 13 22:31:42.283840 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:42.283807 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2fa7da44-h2x7p" Jul 13 22:31:42.294115 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:42.294090 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 13 22:31:42.294263 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:42.294175 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k5z5d\"" Jul 13 22:31:42.312020 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:42.311987 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2fa7da44-h2x7p"] Jul 13 22:31:42.448061 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:42.448025 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/463cd8a3-f1c4-4389-b57e-bc854374fe4c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2fa7da44-h2x7p\" (UID: \"463cd8a3-f1c4-4389-b57e-bc854374fe4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2fa7da44-h2x7p" Jul 13 22:31:42.548661 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:42.548560 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/463cd8a3-f1c4-4389-b57e-bc854374fe4c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2fa7da44-h2x7p\" (UID: \"463cd8a3-f1c4-4389-b57e-bc854374fe4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2fa7da44-h2x7p" Jul 13 22:31:42.549243 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:42.549219 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/463cd8a3-f1c4-4389-b57e-bc854374fe4c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2fa7da44-h2x7p\" (UID: \"463cd8a3-f1c4-4389-b57e-bc854374fe4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2fa7da44-h2x7p" Jul 13 22:31:42.596219 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:42.596177 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2fa7da44-h2x7p" Jul 13 22:31:42.744803 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:42.744778 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2fa7da44-h2x7p"] Jul 13 22:31:42.747337 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:31:42.747302 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod463cd8a3_f1c4_4389_b57e_bc854374fe4c.slice/crio-d2fc6cc07684867dd2bbd7a506192bf00eb57456a4775e4657503e0dc575fc52 WatchSource:0}: Error finding container d2fc6cc07684867dd2bbd7a506192bf00eb57456a4775e4657503e0dc575fc52: Status 404 returned error can't find the container with id d2fc6cc07684867dd2bbd7a506192bf00eb57456a4775e4657503e0dc575fc52 Jul 13 22:31:43.652114 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:43.652083 2555 generic.go:356] "Generic (PLEG): container finished" podID="463cd8a3-f1c4-4389-b57e-bc854374fe4c" containerID="ce99283541019ba2caecd594ab349cdccf6f0bb59c03a7b16d25255791a5d7fc" exitCode=0 Jul 13 22:31:43.653445 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:43.652184 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2fa7da44-h2x7p" event={"ID":"463cd8a3-f1c4-4389-b57e-bc854374fe4c","Type":"ContainerDied","Data":"ce99283541019ba2caecd594ab349cdccf6f0bb59c03a7b16d25255791a5d7fc"} Jul 13 22:31:43.653445 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:43.652268 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2fa7da44-h2x7p" event={"ID":"463cd8a3-f1c4-4389-b57e-bc854374fe4c","Type":"ContainerStarted","Data":"d2fc6cc07684867dd2bbd7a506192bf00eb57456a4775e4657503e0dc575fc52"} Jul 13 22:31:44.562417 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:44.562363 2555 patch_prober.go:28] interesting pod/e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 13 22:31:44.562784 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:44.562737 2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk" podUID="7f94fa89-2178-4ff5-a738-58e4e622ce68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 13 22:31:44.659717 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:44.659683 2555 generic.go:356] "Generic (PLEG): container finished" podID="7f94fa89-2178-4ff5-a738-58e4e622ce68" containerID="df385043694a7e88b47dcdeecb573498d1373dfd065aff07797a569a38de366e" exitCode=0 Jul 13 22:31:44.660106 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:44.659755 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk" event={"ID":"7f94fa89-2178-4ff5-a738-58e4e622ce68","Type":"ContainerDied","Data":"df385043694a7e88b47dcdeecb573498d1373dfd065aff07797a569a38de366e"} Jul 13 22:31:44.660475 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:44.660236 2555 scope.go:117] "RemoveContainer" containerID="df385043694a7e88b47dcdeecb573498d1373dfd065aff07797a569a38de366e" Jul 13 22:31:44.661251 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:44.661220 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:31:44.661364 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:44.661292 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:31:44.661364 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:44.661328 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:31:44.800493 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:44.800453 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2fa7da44-h2x7p" Jul 13 22:31:44.975819 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:44.975772 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/463cd8a3-f1c4-4389-b57e-bc854374fe4c-maas-api-service-ca\") pod \"463cd8a3-f1c4-4389-b57e-bc854374fe4c\" (UID: \"463cd8a3-f1c4-4389-b57e-bc854374fe4c\") " Jul 13 22:31:44.976123 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:44.976103 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/463cd8a3-f1c4-4389-b57e-bc854374fe4c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "463cd8a3-f1c4-4389-b57e-bc854374fe4c" (UID: "463cd8a3-f1c4-4389-b57e-bc854374fe4c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:31:45.076609 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:45.076571 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/463cd8a3-f1c4-4389-b57e-bc854374fe4c-maas-api-service-ca\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:31:45.667909 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:45.667874 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk" event={"ID":"7f94fa89-2178-4ff5-a738-58e4e622ce68","Type":"ContainerStarted","Data":"3b0d962af6fec9b7c475b80d554a217aa6681c15fc65ae190d894f9231014d83"} Jul 13 22:31:45.669148 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:45.669118 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2fa7da44-h2x7p" event={"ID":"463cd8a3-f1c4-4389-b57e-bc854374fe4c","Type":"ContainerDied","Data":"d2fc6cc07684867dd2bbd7a506192bf00eb57456a4775e4657503e0dc575fc52"} Jul 13 22:31:45.669148 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:45.669142 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2fa7da44-h2x7p" Jul 13 22:31:45.669351 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:45.669159 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2fc6cc07684867dd2bbd7a506192bf00eb57456a4775e4657503e0dc575fc52" Jul 13 22:31:46.571718 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:46.571690 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk" Jul 13 22:31:46.571958 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:46.571935 2555 patch_prober.go:28] interesting pod/e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 13 22:31:46.572011 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:46.571986 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk" podUID="7f94fa89-2178-4ff5-a738-58e4e622ce68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 13 22:31:47.572500 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:47.572461 2555 patch_prober.go:28] interesting pod/e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 13 22:31:47.572917 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:47.572520 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk" podUID="7f94fa89-2178-4ff5-a738-58e4e622ce68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 13 22:31:48.572842 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:48.572803 2555 patch_prober.go:28] interesting pod/e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 13 22:31:48.573267 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:48.572861 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk" podUID="7f94fa89-2178-4ff5-a738-58e4e622ce68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 13 22:31:49.576176 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:49.576140 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk" Jul 13 22:31:49.576649 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:49.576426 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk" Jul 13 22:31:49.577254 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:31:49.577237 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk" Jul 13 22:32:07.983536 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:07.983499 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk"] Jul 13 22:32:07.984147 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:07.983752 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk" podUID="7f94fa89-2178-4ff5-a738-58e4e622ce68" containerName="istio-proxy" containerID="cri-o://3b0d962af6fec9b7c475b80d554a217aa6681c15fc65ae190d894f9231014d83" gracePeriod=30 Jul 13 22:32:13.238130 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:13.238103 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk" Jul 13 22:32:13.308798 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:13.308707 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f94fa89-2178-4ff5-a738-58e4e622ce68-istio-data\") pod \"7f94fa89-2178-4ff5-a738-58e4e622ce68\" (UID: \"7f94fa89-2178-4ff5-a738-58e4e622ce68\") " Jul 13 22:32:13.308798 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:13.308770 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f94fa89-2178-4ff5-a738-58e4e622ce68-istio-token\") pod \"7f94fa89-2178-4ff5-a738-58e4e622ce68\" (UID: \"7f94fa89-2178-4ff5-a738-58e4e622ce68\") " Jul 13 22:32:13.309038 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:13.308814 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lq4cg\" (UniqueName: \"kubernetes.io/projected/7f94fa89-2178-4ff5-a738-58e4e622ce68-kube-api-access-lq4cg\") pod \"7f94fa89-2178-4ff5-a738-58e4e622ce68\" (UID: \"7f94fa89-2178-4ff5-a738-58e4e622ce68\") " Jul 13 22:32:13.309038 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:13.308836 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f94fa89-2178-4ff5-a738-58e4e622ce68-istio-envoy\") pod \"7f94fa89-2178-4ff5-a738-58e4e622ce68\" (UID: \"7f94fa89-2178-4ff5-a738-58e4e622ce68\") " Jul 13 22:32:13.309038 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:13.308881 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f94fa89-2178-4ff5-a738-58e4e622ce68-istiod-ca-cert\") pod \"7f94fa89-2178-4ff5-a738-58e4e622ce68\" (UID: \"7f94fa89-2178-4ff5-a738-58e4e622ce68\") " Jul 13 22:32:13.309038 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:13.308917 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f94fa89-2178-4ff5-a738-58e4e622ce68-istio-podinfo\") pod \"7f94fa89-2178-4ff5-a738-58e4e622ce68\" (UID: \"7f94fa89-2178-4ff5-a738-58e4e622ce68\") " Jul 13 22:32:13.309038 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:13.308954 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f94fa89-2178-4ff5-a738-58e4e622ce68-workload-socket\") pod \"7f94fa89-2178-4ff5-a738-58e4e622ce68\" (UID: \"7f94fa89-2178-4ff5-a738-58e4e622ce68\") " Jul 13 22:32:13.309038 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:13.308999 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f94fa89-2178-4ff5-a738-58e4e622ce68-credential-socket\") pod \"7f94fa89-2178-4ff5-a738-58e4e622ce68\" (UID: \"7f94fa89-2178-4ff5-a738-58e4e622ce68\") " Jul 13 22:32:13.309038 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:13.309037 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f94fa89-2178-4ff5-a738-58e4e622ce68-workload-certs\") pod \"7f94fa89-2178-4ff5-a738-58e4e622ce68\" (UID: \"7f94fa89-2178-4ff5-a738-58e4e622ce68\") " Jul 13 22:32:13.309417 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:13.309343 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f94fa89-2178-4ff5-a738-58e4e622ce68-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7f94fa89-2178-4ff5-a738-58e4e622ce68" (UID: "7f94fa89-2178-4ff5-a738-58e4e622ce68"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:32:13.309472 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:13.309413 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f94fa89-2178-4ff5-a738-58e4e622ce68-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7f94fa89-2178-4ff5-a738-58e4e622ce68" (UID: "7f94fa89-2178-4ff5-a738-58e4e622ce68"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:32:13.309524 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:13.309506 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f94fa89-2178-4ff5-a738-58e4e622ce68-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7f94fa89-2178-4ff5-a738-58e4e622ce68" (UID: "7f94fa89-2178-4ff5-a738-58e4e622ce68"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:32:13.309581 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:13.309544 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f94fa89-2178-4ff5-a738-58e4e622ce68-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7f94fa89-2178-4ff5-a738-58e4e622ce68" (UID: "7f94fa89-2178-4ff5-a738-58e4e622ce68"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:32:13.309727 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:13.309704 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f94fa89-2178-4ff5-a738-58e4e622ce68-istio-data" (OuterVolumeSpecName: "istio-data") pod "7f94fa89-2178-4ff5-a738-58e4e622ce68" (UID: "7f94fa89-2178-4ff5-a738-58e4e622ce68"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:32:13.311190 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:13.311162 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f94fa89-2178-4ff5-a738-58e4e622ce68-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7f94fa89-2178-4ff5-a738-58e4e622ce68" (UID: "7f94fa89-2178-4ff5-a738-58e4e622ce68"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:32:13.311475 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:13.311454 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f94fa89-2178-4ff5-a738-58e4e622ce68-istio-token" (OuterVolumeSpecName: "istio-token") pod "7f94fa89-2178-4ff5-a738-58e4e622ce68" (UID: "7f94fa89-2178-4ff5-a738-58e4e622ce68"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:32:13.311704 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:13.311682 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f94fa89-2178-4ff5-a738-58e4e622ce68-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7f94fa89-2178-4ff5-a738-58e4e622ce68" (UID: "7f94fa89-2178-4ff5-a738-58e4e622ce68"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 13 22:32:13.311749 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:13.311687 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f94fa89-2178-4ff5-a738-58e4e622ce68-kube-api-access-lq4cg" (OuterVolumeSpecName: "kube-api-access-lq4cg") pod "7f94fa89-2178-4ff5-a738-58e4e622ce68" (UID: "7f94fa89-2178-4ff5-a738-58e4e622ce68"). InnerVolumeSpecName "kube-api-access-lq4cg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:32:13.410587 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:13.410549 2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f94fa89-2178-4ff5-a738-58e4e622ce68-istiod-ca-cert\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:32:13.410587 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:13.410581 2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f94fa89-2178-4ff5-a738-58e4e622ce68-istio-podinfo\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:32:13.410587 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:13.410592 2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f94fa89-2178-4ff5-a738-58e4e622ce68-workload-socket\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:32:13.410820 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:13.410601 2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f94fa89-2178-4ff5-a738-58e4e622ce68-credential-socket\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:32:13.410820 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:13.410610 2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f94fa89-2178-4ff5-a738-58e4e622ce68-workload-certs\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:32:13.410820 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:13.410619 2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f94fa89-2178-4ff5-a738-58e4e622ce68-istio-data\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:32:13.410820 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:13.410626 2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f94fa89-2178-4ff5-a738-58e4e622ce68-istio-token\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:32:13.410820 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:13.410634 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lq4cg\" (UniqueName: \"kubernetes.io/projected/7f94fa89-2178-4ff5-a738-58e4e622ce68-kube-api-access-lq4cg\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:32:13.410820 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:13.410644 2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f94fa89-2178-4ff5-a738-58e4e622ce68-istio-envoy\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:32:13.845308 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:13.845265 2555 generic.go:356] "Generic (PLEG): container finished" podID="7f94fa89-2178-4ff5-a738-58e4e622ce68" containerID="3b0d962af6fec9b7c475b80d554a217aa6681c15fc65ae190d894f9231014d83" exitCode=0 Jul 13 22:32:13.845483 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:13.845325 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk" Jul 13 22:32:13.845483 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:13.845333 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk" event={"ID":"7f94fa89-2178-4ff5-a738-58e4e622ce68","Type":"ContainerDied","Data":"3b0d962af6fec9b7c475b80d554a217aa6681c15fc65ae190d894f9231014d83"} Jul 13 22:32:13.845483 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:13.845408 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk" event={"ID":"7f94fa89-2178-4ff5-a738-58e4e622ce68","Type":"ContainerDied","Data":"e6bb9ca68d8445f30626c10ea99e6731f4ccb9c3ea1c93430468d7f209646e31"} Jul 13 22:32:13.845483 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:13.845430 2555 scope.go:117] "RemoveContainer" containerID="3b0d962af6fec9b7c475b80d554a217aa6681c15fc65ae190d894f9231014d83" Jul 13 22:32:13.855257 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:13.855238 2555 scope.go:117] "RemoveContainer" containerID="df385043694a7e88b47dcdeecb573498d1373dfd065aff07797a569a38de366e" Jul 13 22:32:13.863768 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:13.863747 2555 scope.go:117] "RemoveContainer" containerID="3b0d962af6fec9b7c475b80d554a217aa6681c15fc65ae190d894f9231014d83" Jul 13 22:32:13.864117 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:32:13.864096 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b0d962af6fec9b7c475b80d554a217aa6681c15fc65ae190d894f9231014d83\": container with ID starting with 3b0d962af6fec9b7c475b80d554a217aa6681c15fc65ae190d894f9231014d83 not found: ID does not exist" containerID="3b0d962af6fec9b7c475b80d554a217aa6681c15fc65ae190d894f9231014d83" Jul 13 22:32:13.864231 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:13.864128 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b0d962af6fec9b7c475b80d554a217aa6681c15fc65ae190d894f9231014d83"} err="failed to get container status \"3b0d962af6fec9b7c475b80d554a217aa6681c15fc65ae190d894f9231014d83\": rpc error: code = NotFound desc = could not find container \"3b0d962af6fec9b7c475b80d554a217aa6681c15fc65ae190d894f9231014d83\": container with ID starting with 3b0d962af6fec9b7c475b80d554a217aa6681c15fc65ae190d894f9231014d83 not found: ID does not exist" Jul 13 22:32:13.864231 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:13.864150 2555 scope.go:117] "RemoveContainer" containerID="df385043694a7e88b47dcdeecb573498d1373dfd065aff07797a569a38de366e" Jul 13 22:32:13.864445 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:32:13.864429 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df385043694a7e88b47dcdeecb573498d1373dfd065aff07797a569a38de366e\": container with ID starting with df385043694a7e88b47dcdeecb573498d1373dfd065aff07797a569a38de366e not found: ID does not exist" containerID="df385043694a7e88b47dcdeecb573498d1373dfd065aff07797a569a38de366e" Jul 13 22:32:13.864497 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:13.864454 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df385043694a7e88b47dcdeecb573498d1373dfd065aff07797a569a38de366e"} err="failed to get container status \"df385043694a7e88b47dcdeecb573498d1373dfd065aff07797a569a38de366e\": rpc error: code = NotFound desc = could not find container \"df385043694a7e88b47dcdeecb573498d1373dfd065aff07797a569a38de366e\": container with ID starting with df385043694a7e88b47dcdeecb573498d1373dfd065aff07797a569a38de366e not found: ID does not exist" Jul 13 22:32:13.867052 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:13.867032 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk"] Jul 13 22:32:13.870601 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:13.870583 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-2fa7da44-openshift-default-7565b4f987-v2qkk"] Jul 13 22:32:14.560061 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:14.560021 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f94fa89-2178-4ff5-a738-58e4e622ce68" path="/var/lib/kubelet/pods/7f94fa89-2178-4ff5-a738-58e4e622ce68/volumes" Jul 13 22:32:27.214551 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:27.214430 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv"] Jul 13 22:32:27.216226 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:27.216154 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="463cd8a3-f1c4-4389-b57e-bc854374fe4c" containerName="revoke-keys" Jul 13 22:32:27.216226 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:27.216183 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="463cd8a3-f1c4-4389-b57e-bc854374fe4c" containerName="revoke-keys" Jul 13 22:32:27.216538 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:27.216483 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f94fa89-2178-4ff5-a738-58e4e622ce68" containerName="istio-proxy" Jul 13 22:32:27.216538 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:27.216501 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f94fa89-2178-4ff5-a738-58e4e622ce68" containerName="istio-proxy" Jul 13 22:32:27.217010 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:27.216955 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f94fa89-2178-4ff5-a738-58e4e622ce68" containerName="istio-proxy" Jul 13 22:32:27.217010 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:27.216980 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="463cd8a3-f1c4-4389-b57e-bc854374fe4c" containerName="revoke-keys" Jul 13 22:32:27.221960 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:27.221933 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv" Jul 13 22:32:27.224973 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:27.224951 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-86e22e9e-openshift-default-dockercfg-vxg2h\"" Jul 13 22:32:27.232032 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:27.232007 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv"] Jul 13 22:32:27.337530 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:27.337491 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17eb4d3d-212c-4faf-87cf-960879e5c333-istiod-ca-cert\") pod \"e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv\" (UID: \"17eb4d3d-212c-4faf-87cf-960879e5c333\") " pod="openshift-ingress/e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv" Jul 13 22:32:27.337718 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:27.337563 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17eb4d3d-212c-4faf-87cf-960879e5c333-istio-envoy\") pod \"e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv\" (UID: \"17eb4d3d-212c-4faf-87cf-960879e5c333\") " pod="openshift-ingress/e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv" Jul 13 22:32:27.337718 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:27.337696 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn4vx\" (UniqueName: \"kubernetes.io/projected/17eb4d3d-212c-4faf-87cf-960879e5c333-kube-api-access-bn4vx\") pod \"e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv\" (UID: \"17eb4d3d-212c-4faf-87cf-960879e5c333\") " pod="openshift-ingress/e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv" Jul 13 22:32:27.337816 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:27.337751 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17eb4d3d-212c-4faf-87cf-960879e5c333-workload-certs\") pod \"e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv\" (UID: \"17eb4d3d-212c-4faf-87cf-960879e5c333\") " pod="openshift-ingress/e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv" Jul 13 22:32:27.337894 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:27.337866 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17eb4d3d-212c-4faf-87cf-960879e5c333-credential-socket\") pod \"e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv\" (UID: \"17eb4d3d-212c-4faf-87cf-960879e5c333\") " pod="openshift-ingress/e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv" Jul 13 22:32:27.337955 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:27.337922 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17eb4d3d-212c-4faf-87cf-960879e5c333-workload-socket\") pod \"e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv\" (UID: \"17eb4d3d-212c-4faf-87cf-960879e5c333\") " pod="openshift-ingress/e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv" Jul 13 22:32:27.338011 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:27.337950 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17eb4d3d-212c-4faf-87cf-960879e5c333-istio-podinfo\") pod \"e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv\" (UID: \"17eb4d3d-212c-4faf-87cf-960879e5c333\") " pod="openshift-ingress/e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv" Jul 13 22:32:27.338011 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:27.337979 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17eb4d3d-212c-4faf-87cf-960879e5c333-istio-token\") pod \"e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv\" (UID: \"17eb4d3d-212c-4faf-87cf-960879e5c333\") " pod="openshift-ingress/e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv" Jul 13 22:32:27.338108 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:27.338012 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17eb4d3d-212c-4faf-87cf-960879e5c333-istio-data\") pod \"e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv\" (UID: \"17eb4d3d-212c-4faf-87cf-960879e5c333\") " pod="openshift-ingress/e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv" Jul 13 22:32:27.439087 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:27.439044 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17eb4d3d-212c-4faf-87cf-960879e5c333-credential-socket\") pod \"e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv\" (UID: \"17eb4d3d-212c-4faf-87cf-960879e5c333\") " pod="openshift-ingress/e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv" Jul 13 22:32:27.439087 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:27.439091 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17eb4d3d-212c-4faf-87cf-960879e5c333-workload-socket\") pod \"e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv\" (UID: \"17eb4d3d-212c-4faf-87cf-960879e5c333\") " pod="openshift-ingress/e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv" Jul 13 22:32:27.439360 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:27.439107 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17eb4d3d-212c-4faf-87cf-960879e5c333-istio-podinfo\") pod \"e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv\" (UID: \"17eb4d3d-212c-4faf-87cf-960879e5c333\") " pod="openshift-ingress/e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv" Jul 13 22:32:27.439360 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:27.439134 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17eb4d3d-212c-4faf-87cf-960879e5c333-istio-token\") pod \"e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv\" (UID: \"17eb4d3d-212c-4faf-87cf-960879e5c333\") " pod="openshift-ingress/e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv" Jul 13 22:32:27.439360 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:27.439164 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17eb4d3d-212c-4faf-87cf-960879e5c333-istio-data\") pod \"e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv\" (UID: \"17eb4d3d-212c-4faf-87cf-960879e5c333\") " pod="openshift-ingress/e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv" Jul 13 22:32:27.439553 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:27.439526 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17eb4d3d-212c-4faf-87cf-960879e5c333-workload-socket\") pod \"e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv\" (UID: \"17eb4d3d-212c-4faf-87cf-960879e5c333\") " pod="openshift-ingress/e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv" Jul 13 22:32:27.439631 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:27.439217 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17eb4d3d-212c-4faf-87cf-960879e5c333-istiod-ca-cert\") pod \"e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv\" (UID: \"17eb4d3d-212c-4faf-87cf-960879e5c333\") " pod="openshift-ingress/e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv" Jul 13 22:32:27.439775 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:27.439750 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17eb4d3d-212c-4faf-87cf-960879e5c333-istio-data\") pod \"e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv\" (UID: \"17eb4d3d-212c-4faf-87cf-960879e5c333\") " pod="openshift-ingress/e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv" Jul 13 22:32:27.439850 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:27.439689 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17eb4d3d-212c-4faf-87cf-960879e5c333-credential-socket\") pod \"e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv\" (UID: \"17eb4d3d-212c-4faf-87cf-960879e5c333\") " pod="openshift-ingress/e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv" Jul 13 22:32:27.439850 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:27.439790 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17eb4d3d-212c-4faf-87cf-960879e5c333-istio-envoy\") pod \"e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv\" (UID: \"17eb4d3d-212c-4faf-87cf-960879e5c333\") " pod="openshift-ingress/e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv" Jul 13 22:32:27.439953 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:27.439854 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn4vx\" (UniqueName: \"kubernetes.io/projected/17eb4d3d-212c-4faf-87cf-960879e5c333-kube-api-access-bn4vx\") pod \"e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv\" (UID: \"17eb4d3d-212c-4faf-87cf-960879e5c333\") " pod="openshift-ingress/e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv" Jul 13 22:32:27.439953 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:27.439896 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17eb4d3d-212c-4faf-87cf-960879e5c333-workload-certs\") pod \"e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv\" (UID: \"17eb4d3d-212c-4faf-87cf-960879e5c333\") " pod="openshift-ingress/e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv" Jul 13 22:32:27.440234 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:27.440193 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17eb4d3d-212c-4faf-87cf-960879e5c333-workload-certs\") pod \"e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv\" (UID: \"17eb4d3d-212c-4faf-87cf-960879e5c333\") " pod="openshift-ingress/e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv" Jul 13 22:32:27.440384 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:27.440283 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17eb4d3d-212c-4faf-87cf-960879e5c333-istiod-ca-cert\") pod \"e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv\" (UID: \"17eb4d3d-212c-4faf-87cf-960879e5c333\") " pod="openshift-ingress/e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv" Jul 13 22:32:27.442102 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:27.442083 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17eb4d3d-212c-4faf-87cf-960879e5c333-istio-envoy\") pod \"e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv\" (UID: \"17eb4d3d-212c-4faf-87cf-960879e5c333\") " pod="openshift-ingress/e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv" Jul 13 22:32:27.442517 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:27.442498 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17eb4d3d-212c-4faf-87cf-960879e5c333-istio-podinfo\") pod \"e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv\" (UID: \"17eb4d3d-212c-4faf-87cf-960879e5c333\") " pod="openshift-ingress/e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv" Jul 13 22:32:27.448641 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:27.448614 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17eb4d3d-212c-4faf-87cf-960879e5c333-istio-token\") pod \"e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv\" (UID: \"17eb4d3d-212c-4faf-87cf-960879e5c333\") " pod="openshift-ingress/e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv" Jul 13 22:32:27.448993 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:27.448968 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn4vx\" (UniqueName: \"kubernetes.io/projected/17eb4d3d-212c-4faf-87cf-960879e5c333-kube-api-access-bn4vx\") pod \"e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv\" (UID: \"17eb4d3d-212c-4faf-87cf-960879e5c333\") " pod="openshift-ingress/e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv" Jul 13 22:32:27.536943 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:27.536855 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv" Jul 13 22:32:27.698654 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:27.698628 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv"] Jul 13 22:32:27.700940 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:32:27.700903 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17eb4d3d_212c_4faf_87cf_960879e5c333.slice/crio-c9afcbb89f403084c5cf1c5a80e10b601a168a49fadb82495e27e2a1cced6fcd WatchSource:0}: Error finding container c9afcbb89f403084c5cf1c5a80e10b601a168a49fadb82495e27e2a1cced6fcd: Status 404 returned error can't find the container with id c9afcbb89f403084c5cf1c5a80e10b601a168a49fadb82495e27e2a1cced6fcd Jul 13 22:32:27.703431 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:27.703390 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:32:27.703519 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:27.703476 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:32:27.703556 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:27.703519 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:32:27.933520 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:27.933482 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv" event={"ID":"17eb4d3d-212c-4faf-87cf-960879e5c333","Type":"ContainerStarted","Data":"49eda9d07ef15c4e1cc24249f94d276059cb68657f66e29873f0bc199e273dbf"} Jul 13 22:32:27.933703 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:27.933549 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv" event={"ID":"17eb4d3d-212c-4faf-87cf-960879e5c333","Type":"ContainerStarted","Data":"c9afcbb89f403084c5cf1c5a80e10b601a168a49fadb82495e27e2a1cced6fcd"} Jul 13 22:32:27.959448 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:27.959383 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv" podStartSLOduration=0.959361693 podStartE2EDuration="959.361693ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:32:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:32:27.954902576 +0000 UTC m=+1953.948656905" watchObservedRunningTime="2026-07-13 22:32:27.959361693 +0000 UTC m=+1953.953115987" Jul 13 22:32:28.537272 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:28.537235 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv" Jul 13 22:32:28.542150 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:28.542126 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv" Jul 13 22:32:28.939739 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:28.939701 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv" Jul 13 22:32:28.940846 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:28.940824 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv" Jul 13 22:32:43.029902 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:43.029867 2555 generic.go:356] "Generic (PLEG): container finished" podID="17eb4d3d-212c-4faf-87cf-960879e5c333" containerID="49eda9d07ef15c4e1cc24249f94d276059cb68657f66e29873f0bc199e273dbf" exitCode=0 Jul 13 22:32:43.030327 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:43.029943 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv" event={"ID":"17eb4d3d-212c-4faf-87cf-960879e5c333","Type":"ContainerDied","Data":"49eda9d07ef15c4e1cc24249f94d276059cb68657f66e29873f0bc199e273dbf"} Jul 13 22:32:43.030392 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:43.030373 2555 scope.go:117] "RemoveContainer" containerID="49eda9d07ef15c4e1cc24249f94d276059cb68657f66e29873f0bc199e273dbf" Jul 13 22:32:43.031394 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:43.031365 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:32:43.031502 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:43.031432 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:32:43.031502 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:43.031465 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:32:43.939879 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:43.939845 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv" Jul 13 22:32:44.037321 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:44.037276 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv" event={"ID":"17eb4d3d-212c-4faf-87cf-960879e5c333","Type":"ContainerStarted","Data":"50467d42895c65382a21fcbf341e4b618261608765dafb8b72ae6f8bcfe71ef7"} Jul 13 22:32:44.538025 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:44.537991 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv" Jul 13 22:32:44.539150 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:44.539122 2555 patch_prober.go:28] interesting pod/e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Jul 13 22:32:44.539266 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:44.539246 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv" podUID="17eb4d3d-212c-4faf-87cf-960879e5c333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Jul 13 22:32:45.538177 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:45.538145 2555 patch_prober.go:28] interesting pod/e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Jul 13 22:32:45.538592 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:45.538226 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv" podUID="17eb4d3d-212c-4faf-87cf-960879e5c333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Jul 13 22:32:46.537806 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:46.537771 2555 patch_prober.go:28] interesting pod/e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Jul 13 22:32:46.537992 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:46.537827 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv" podUID="17eb4d3d-212c-4faf-87cf-960879e5c333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Jul 13 22:32:47.537814 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:47.537778 2555 patch_prober.go:28] interesting pod/e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Jul 13 22:32:47.538289 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:47.537841 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv" podUID="17eb4d3d-212c-4faf-87cf-960879e5c333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Jul 13 22:32:48.541160 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:48.541128 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv" Jul 13 22:32:48.541568 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:48.541407 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv" Jul 13 22:32:48.542114 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:48.542097 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv" Jul 13 22:32:54.945882 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:32:54.945844 2555 scope.go:117] "RemoveContainer" containerID="bbd6c0e5171d02427dffa4c69559629d9b0ea94390f3420a443afae269e85562" Jul 13 22:33:17.599484 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:33:17.599440 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-86e22e9e-9dj4r"] Jul 13 22:33:17.600035 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:33:17.599880 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f94fa89-2178-4ff5-a738-58e4e622ce68" containerName="istio-proxy" Jul 13 22:33:17.600035 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:33:17.599897 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f94fa89-2178-4ff5-a738-58e4e622ce68" containerName="istio-proxy" Jul 13 22:33:17.600035 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:33:17.599978 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f94fa89-2178-4ff5-a738-58e4e622ce68" containerName="istio-proxy" Jul 13 22:33:17.603097 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:33:17.603076 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-86e22e9e-9dj4r" Jul 13 22:33:17.609229 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:33:17.606130 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 13 22:33:17.609229 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:33:17.606250 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k5z5d\"" Jul 13 22:33:17.612820 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:33:17.612791 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-86e22e9e-9dj4r"] Jul 13 22:33:17.715763 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:33:17.715720 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1010f23b-7bcf-4a62-bd05-f070208d24c2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-86e22e9e-9dj4r\" (UID: \"1010f23b-7bcf-4a62-bd05-f070208d24c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-86e22e9e-9dj4r" Jul 13 22:33:17.816991 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:33:17.816951 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1010f23b-7bcf-4a62-bd05-f070208d24c2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-86e22e9e-9dj4r\" (UID: \"1010f23b-7bcf-4a62-bd05-f070208d24c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-86e22e9e-9dj4r" Jul 13 22:33:17.817569 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:33:17.817550 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1010f23b-7bcf-4a62-bd05-f070208d24c2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-86e22e9e-9dj4r\" (UID: \"1010f23b-7bcf-4a62-bd05-f070208d24c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-86e22e9e-9dj4r" Jul 13 22:33:17.920354 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:33:17.920318 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-86e22e9e-9dj4r" Jul 13 22:33:18.059058 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:33:18.059027 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-86e22e9e-9dj4r"] Jul 13 22:33:18.059976 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:33:18.059953 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1010f23b_7bcf_4a62_bd05_f070208d24c2.slice/crio-4d2134a34985a15330080807eae3f99b065d67efd4a2c2096409d14d0c0e9fc8 WatchSource:0}: Error finding container 4d2134a34985a15330080807eae3f99b065d67efd4a2c2096409d14d0c0e9fc8: Status 404 returned error can't find the container with id 4d2134a34985a15330080807eae3f99b065d67efd4a2c2096409d14d0c0e9fc8 Jul 13 22:33:18.233058 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:33:18.232971 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-86e22e9e-9dj4r" event={"ID":"1010f23b-7bcf-4a62-bd05-f070208d24c2","Type":"ContainerStarted","Data":"4d2134a34985a15330080807eae3f99b065d67efd4a2c2096409d14d0c0e9fc8"} Jul 13 22:33:19.239953 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:33:19.239916 2555 generic.go:356] "Generic (PLEG): container finished" podID="1010f23b-7bcf-4a62-bd05-f070208d24c2" containerID="428589255c2b2ea5a6ad1268c69e82d9da071da9798b2dfbb68a01242b864c9f" exitCode=0 Jul 13 22:33:19.240350 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:33:19.239965 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-86e22e9e-9dj4r" event={"ID":"1010f23b-7bcf-4a62-bd05-f070208d24c2","Type":"ContainerDied","Data":"428589255c2b2ea5a6ad1268c69e82d9da071da9798b2dfbb68a01242b864c9f"} Jul 13 22:33:20.376992 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:33:20.376966 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-86e22e9e-9dj4r" Jul 13 22:33:20.540291 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:33:20.540166 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1010f23b-7bcf-4a62-bd05-f070208d24c2-maas-api-service-ca\") pod \"1010f23b-7bcf-4a62-bd05-f070208d24c2\" (UID: \"1010f23b-7bcf-4a62-bd05-f070208d24c2\") " Jul 13 22:33:20.540610 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:33:20.540581 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1010f23b-7bcf-4a62-bd05-f070208d24c2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1010f23b-7bcf-4a62-bd05-f070208d24c2" (UID: "1010f23b-7bcf-4a62-bd05-f070208d24c2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:33:20.641163 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:33:20.641123 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1010f23b-7bcf-4a62-bd05-f070208d24c2-maas-api-service-ca\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:33:21.252693 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:33:21.252659 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-86e22e9e-9dj4r" Jul 13 22:33:21.252693 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:33:21.252685 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-86e22e9e-9dj4r" event={"ID":"1010f23b-7bcf-4a62-bd05-f070208d24c2","Type":"ContainerDied","Data":"4d2134a34985a15330080807eae3f99b065d67efd4a2c2096409d14d0c0e9fc8"} Jul 13 22:33:21.252967 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:33:21.252728 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d2134a34985a15330080807eae3f99b065d67efd4a2c2096409d14d0c0e9fc8" Jul 13 22:33:54.974293 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:33:54.974254 2555 scope.go:117] "RemoveContainer" containerID="ce0a64eef6fb398912265571744000af7a58e2db4878eb967d4e41d6c978ff5e" Jul 13 22:34:44.898595 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:34:44.898564 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv"] Jul 13 22:34:44.899164 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:34:44.898826 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv" podUID="17eb4d3d-212c-4faf-87cf-960879e5c333" containerName="istio-proxy" containerID="cri-o://50467d42895c65382a21fcbf341e4b618261608765dafb8b72ae6f8bcfe71ef7" gracePeriod=30 Jul 13 22:34:48.544226 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:34:48.544175 2555 patch_prober.go:28] interesting pod/e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 13 22:34:48.544601 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:34:48.544259 2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv" podUID="17eb4d3d-212c-4faf-87cf-960879e5c333" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 13 22:34:50.163620 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:34:50.163538 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv" Jul 13 22:34:50.330677 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:34:50.330637 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17eb4d3d-212c-4faf-87cf-960879e5c333-istio-podinfo\") pod \"17eb4d3d-212c-4faf-87cf-960879e5c333\" (UID: \"17eb4d3d-212c-4faf-87cf-960879e5c333\") " Jul 13 22:34:50.330878 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:34:50.330704 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17eb4d3d-212c-4faf-87cf-960879e5c333-istiod-ca-cert\") pod \"17eb4d3d-212c-4faf-87cf-960879e5c333\" (UID: \"17eb4d3d-212c-4faf-87cf-960879e5c333\") " Jul 13 22:34:50.330878 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:34:50.330812 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17eb4d3d-212c-4faf-87cf-960879e5c333-istio-data\") pod \"17eb4d3d-212c-4faf-87cf-960879e5c333\" (UID: \"17eb4d3d-212c-4faf-87cf-960879e5c333\") " Jul 13 22:34:50.331010 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:34:50.330891 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17eb4d3d-212c-4faf-87cf-960879e5c333-istio-envoy\") pod \"17eb4d3d-212c-4faf-87cf-960879e5c333\" (UID: \"17eb4d3d-212c-4faf-87cf-960879e5c333\") " Jul 13 22:34:50.331010 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:34:50.330921 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17eb4d3d-212c-4faf-87cf-960879e5c333-istio-token\") pod \"17eb4d3d-212c-4faf-87cf-960879e5c333\" (UID: \"17eb4d3d-212c-4faf-87cf-960879e5c333\") " Jul 13 22:34:50.331010 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:34:50.330952 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bn4vx\" (UniqueName: \"kubernetes.io/projected/17eb4d3d-212c-4faf-87cf-960879e5c333-kube-api-access-bn4vx\") pod \"17eb4d3d-212c-4faf-87cf-960879e5c333\" (UID: \"17eb4d3d-212c-4faf-87cf-960879e5c333\") " Jul 13 22:34:50.331164 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:34:50.331012 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17eb4d3d-212c-4faf-87cf-960879e5c333-workload-certs\") pod \"17eb4d3d-212c-4faf-87cf-960879e5c333\" (UID: \"17eb4d3d-212c-4faf-87cf-960879e5c333\") " Jul 13 22:34:50.331164 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:34:50.331036 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17eb4d3d-212c-4faf-87cf-960879e5c333-credential-socket\") pod \"17eb4d3d-212c-4faf-87cf-960879e5c333\" (UID: \"17eb4d3d-212c-4faf-87cf-960879e5c333\") " Jul 13 22:34:50.331164 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:34:50.331067 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17eb4d3d-212c-4faf-87cf-960879e5c333-workload-socket\") pod \"17eb4d3d-212c-4faf-87cf-960879e5c333\" (UID: \"17eb4d3d-212c-4faf-87cf-960879e5c333\") " Jul 13 22:34:50.331164 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:34:50.331083 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17eb4d3d-212c-4faf-87cf-960879e5c333-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "17eb4d3d-212c-4faf-87cf-960879e5c333" (UID: "17eb4d3d-212c-4faf-87cf-960879e5c333"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:34:50.331408 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:34:50.331382 2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17eb4d3d-212c-4faf-87cf-960879e5c333-istiod-ca-cert\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:34:50.331715 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:34:50.331545 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17eb4d3d-212c-4faf-87cf-960879e5c333-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "17eb4d3d-212c-4faf-87cf-960879e5c333" (UID: "17eb4d3d-212c-4faf-87cf-960879e5c333"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:34:50.331715 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:34:50.331593 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17eb4d3d-212c-4faf-87cf-960879e5c333-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "17eb4d3d-212c-4faf-87cf-960879e5c333" (UID: "17eb4d3d-212c-4faf-87cf-960879e5c333"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:34:50.331715 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:34:50.331687 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17eb4d3d-212c-4faf-87cf-960879e5c333-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "17eb4d3d-212c-4faf-87cf-960879e5c333" (UID: "17eb4d3d-212c-4faf-87cf-960879e5c333"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:34:50.332195 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:34:50.332154 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17eb4d3d-212c-4faf-87cf-960879e5c333-istio-data" (OuterVolumeSpecName: "istio-data") pod "17eb4d3d-212c-4faf-87cf-960879e5c333" (UID: "17eb4d3d-212c-4faf-87cf-960879e5c333"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:34:50.333390 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:34:50.333366 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17eb4d3d-212c-4faf-87cf-960879e5c333-kube-api-access-bn4vx" (OuterVolumeSpecName: "kube-api-access-bn4vx") pod "17eb4d3d-212c-4faf-87cf-960879e5c333" (UID: "17eb4d3d-212c-4faf-87cf-960879e5c333"). InnerVolumeSpecName "kube-api-access-bn4vx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:34:50.334075 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:34:50.334039 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17eb4d3d-212c-4faf-87cf-960879e5c333-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "17eb4d3d-212c-4faf-87cf-960879e5c333" (UID: "17eb4d3d-212c-4faf-87cf-960879e5c333"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 13 22:34:50.334179 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:34:50.334147 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17eb4d3d-212c-4faf-87cf-960879e5c333-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "17eb4d3d-212c-4faf-87cf-960879e5c333" (UID: "17eb4d3d-212c-4faf-87cf-960879e5c333"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:34:50.334262 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:34:50.334221 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17eb4d3d-212c-4faf-87cf-960879e5c333-istio-token" (OuterVolumeSpecName: "istio-token") pod "17eb4d3d-212c-4faf-87cf-960879e5c333" (UID: "17eb4d3d-212c-4faf-87cf-960879e5c333"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:34:50.432265 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:34:50.432141 2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17eb4d3d-212c-4faf-87cf-960879e5c333-istio-podinfo\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:34:50.432265 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:34:50.432184 2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17eb4d3d-212c-4faf-87cf-960879e5c333-istio-data\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:34:50.432265 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:34:50.432232 2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17eb4d3d-212c-4faf-87cf-960879e5c333-istio-envoy\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:34:50.432265 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:34:50.432247 2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17eb4d3d-212c-4faf-87cf-960879e5c333-istio-token\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:34:50.432265 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:34:50.432261 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn4vx\" (UniqueName: \"kubernetes.io/projected/17eb4d3d-212c-4faf-87cf-960879e5c333-kube-api-access-bn4vx\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:34:50.432679 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:34:50.432275 2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17eb4d3d-212c-4faf-87cf-960879e5c333-workload-certs\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:34:50.432679 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:34:50.432289 2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17eb4d3d-212c-4faf-87cf-960879e5c333-credential-socket\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:34:50.432679 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:34:50.432302 2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17eb4d3d-212c-4faf-87cf-960879e5c333-workload-socket\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:34:50.773334 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:34:50.773239 2555 generic.go:356] "Generic (PLEG): container finished" podID="17eb4d3d-212c-4faf-87cf-960879e5c333" containerID="50467d42895c65382a21fcbf341e4b618261608765dafb8b72ae6f8bcfe71ef7" exitCode=0 Jul 13 22:34:50.773510 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:34:50.773330 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv" event={"ID":"17eb4d3d-212c-4faf-87cf-960879e5c333","Type":"ContainerDied","Data":"50467d42895c65382a21fcbf341e4b618261608765dafb8b72ae6f8bcfe71ef7"} Jul 13 22:34:50.773510 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:34:50.773347 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv" Jul 13 22:34:50.773510 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:34:50.773388 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv" event={"ID":"17eb4d3d-212c-4faf-87cf-960879e5c333","Type":"ContainerDied","Data":"c9afcbb89f403084c5cf1c5a80e10b601a168a49fadb82495e27e2a1cced6fcd"} Jul 13 22:34:50.773510 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:34:50.773417 2555 scope.go:117] "RemoveContainer" containerID="50467d42895c65382a21fcbf341e4b618261608765dafb8b72ae6f8bcfe71ef7" Jul 13 22:34:50.783647 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:34:50.783619 2555 scope.go:117] "RemoveContainer" containerID="49eda9d07ef15c4e1cc24249f94d276059cb68657f66e29873f0bc199e273dbf" Jul 13 22:34:50.792944 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:34:50.792923 2555 scope.go:117] "RemoveContainer" containerID="50467d42895c65382a21fcbf341e4b618261608765dafb8b72ae6f8bcfe71ef7" Jul 13 22:34:50.793289 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:34:50.793269 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50467d42895c65382a21fcbf341e4b618261608765dafb8b72ae6f8bcfe71ef7\": container with ID starting with 50467d42895c65382a21fcbf341e4b618261608765dafb8b72ae6f8bcfe71ef7 not found: ID does not exist" containerID="50467d42895c65382a21fcbf341e4b618261608765dafb8b72ae6f8bcfe71ef7" Jul 13 22:34:50.793361 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:34:50.793299 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50467d42895c65382a21fcbf341e4b618261608765dafb8b72ae6f8bcfe71ef7"} err="failed to get container status \"50467d42895c65382a21fcbf341e4b618261608765dafb8b72ae6f8bcfe71ef7\": rpc error: code = NotFound desc = could not find container \"50467d42895c65382a21fcbf341e4b618261608765dafb8b72ae6f8bcfe71ef7\": container with ID starting with 50467d42895c65382a21fcbf341e4b618261608765dafb8b72ae6f8bcfe71ef7 not found: ID does not exist" Jul 13 22:34:50.793361 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:34:50.793319 2555 scope.go:117] "RemoveContainer" containerID="49eda9d07ef15c4e1cc24249f94d276059cb68657f66e29873f0bc199e273dbf" Jul 13 22:34:50.793627 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:34:50.793599 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49eda9d07ef15c4e1cc24249f94d276059cb68657f66e29873f0bc199e273dbf\": container with ID starting with 49eda9d07ef15c4e1cc24249f94d276059cb68657f66e29873f0bc199e273dbf not found: ID does not exist" containerID="49eda9d07ef15c4e1cc24249f94d276059cb68657f66e29873f0bc199e273dbf" Jul 13 22:34:50.793676 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:34:50.793633 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49eda9d07ef15c4e1cc24249f94d276059cb68657f66e29873f0bc199e273dbf"} err="failed to get container status \"49eda9d07ef15c4e1cc24249f94d276059cb68657f66e29873f0bc199e273dbf\": rpc error: code = NotFound desc = could not find container \"49eda9d07ef15c4e1cc24249f94d276059cb68657f66e29873f0bc199e273dbf\": container with ID starting with 49eda9d07ef15c4e1cc24249f94d276059cb68657f66e29873f0bc199e273dbf not found: ID does not exist" Jul 13 22:34:50.808880 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:34:50.808851 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv"] Jul 13 22:34:50.820594 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:34:50.820556 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-86e22e9e-openshift-default-6b8797d4f5-nn9mv"] Jul 13 22:34:52.559943 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:34:52.559909 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17eb4d3d-212c-4faf-87cf-960879e5c333" path="/var/lib/kubelet/pods/17eb4d3d-212c-4faf-87cf-960879e5c333/volumes" Jul 13 22:34:54.995815 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:34:54.995781 2555 scope.go:117] "RemoveContainer" containerID="0dd71d1580bfaca9e675395c7b5a706ff00fe73594e1bbbdc90305960279414f" Jul 13 22:35:39.876412 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:39.876377 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq"] Jul 13 22:35:39.876872 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:39.876766 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17eb4d3d-212c-4faf-87cf-960879e5c333" containerName="istio-proxy" Jul 13 22:35:39.876872 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:39.876779 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="17eb4d3d-212c-4faf-87cf-960879e5c333" containerName="istio-proxy" Jul 13 22:35:39.876872 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:39.876792 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17eb4d3d-212c-4faf-87cf-960879e5c333" containerName="istio-proxy" Jul 13 22:35:39.876872 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:39.876797 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="17eb4d3d-212c-4faf-87cf-960879e5c333" containerName="istio-proxy" Jul 13 22:35:39.876872 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:39.876805 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1010f23b-7bcf-4a62-bd05-f070208d24c2" containerName="revoke-keys" Jul 13 22:35:39.876872 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:39.876811 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="1010f23b-7bcf-4a62-bd05-f070208d24c2" containerName="revoke-keys" Jul 13 22:35:39.876872 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:39.876874 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="1010f23b-7bcf-4a62-bd05-f070208d24c2" containerName="revoke-keys" Jul 13 22:35:39.877087 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:39.876883 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="17eb4d3d-212c-4faf-87cf-960879e5c333" containerName="istio-proxy" Jul 13 22:35:39.879861 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:39.879843 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq" Jul 13 22:35:39.888973 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:39.888954 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-7da020-openshift-default-dockercfg-ns2vt\"" Jul 13 22:35:39.932097 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:39.932067 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq"] Jul 13 22:35:39.944520 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:39.944487 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/911216b4-9bcb-4ef4-9a70-eba330c4a29b-credential-socket\") pod \"e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq\" (UID: \"911216b4-9bcb-4ef4-9a70-eba330c4a29b\") " pod="openshift-ingress/e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq" Jul 13 22:35:39.944657 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:39.944541 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/911216b4-9bcb-4ef4-9a70-eba330c4a29b-workload-certs\") pod \"e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq\" (UID: \"911216b4-9bcb-4ef4-9a70-eba330c4a29b\") " pod="openshift-ingress/e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq" Jul 13 22:35:39.944657 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:39.944615 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/911216b4-9bcb-4ef4-9a70-eba330c4a29b-istio-podinfo\") pod \"e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq\" (UID: \"911216b4-9bcb-4ef4-9a70-eba330c4a29b\") " pod="openshift-ingress/e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq" Jul 13 22:35:39.944735 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:39.944660 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/911216b4-9bcb-4ef4-9a70-eba330c4a29b-istio-envoy\") pod \"e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq\" (UID: \"911216b4-9bcb-4ef4-9a70-eba330c4a29b\") " pod="openshift-ingress/e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq" Jul 13 22:35:39.944735 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:39.944676 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/911216b4-9bcb-4ef4-9a70-eba330c4a29b-istio-token\") pod \"e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq\" (UID: \"911216b4-9bcb-4ef4-9a70-eba330c4a29b\") " pod="openshift-ingress/e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq" Jul 13 22:35:39.944735 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:39.944704 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/911216b4-9bcb-4ef4-9a70-eba330c4a29b-istio-data\") pod \"e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq\" (UID: \"911216b4-9bcb-4ef4-9a70-eba330c4a29b\") " pod="openshift-ingress/e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq" Jul 13 22:35:39.944735 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:39.944730 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/911216b4-9bcb-4ef4-9a70-eba330c4a29b-workload-socket\") pod \"e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq\" (UID: \"911216b4-9bcb-4ef4-9a70-eba330c4a29b\") " pod="openshift-ingress/e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq" Jul 13 22:35:39.944866 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:39.944767 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/911216b4-9bcb-4ef4-9a70-eba330c4a29b-istiod-ca-cert\") pod \"e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq\" (UID: \"911216b4-9bcb-4ef4-9a70-eba330c4a29b\") " pod="openshift-ingress/e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq" Jul 13 22:35:39.944866 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:39.944783 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7bhw\" (UniqueName: \"kubernetes.io/projected/911216b4-9bcb-4ef4-9a70-eba330c4a29b-kube-api-access-j7bhw\") pod \"e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq\" (UID: \"911216b4-9bcb-4ef4-9a70-eba330c4a29b\") " pod="openshift-ingress/e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq" Jul 13 22:35:40.045347 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:40.045300 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/911216b4-9bcb-4ef4-9a70-eba330c4a29b-credential-socket\") pod \"e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq\" (UID: \"911216b4-9bcb-4ef4-9a70-eba330c4a29b\") " pod="openshift-ingress/e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq" Jul 13 22:35:40.045541 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:40.045370 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/911216b4-9bcb-4ef4-9a70-eba330c4a29b-workload-certs\") pod \"e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq\" (UID: \"911216b4-9bcb-4ef4-9a70-eba330c4a29b\") " pod="openshift-ingress/e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq" Jul 13 22:35:40.045541 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:40.045410 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/911216b4-9bcb-4ef4-9a70-eba330c4a29b-istio-podinfo\") pod \"e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq\" (UID: \"911216b4-9bcb-4ef4-9a70-eba330c4a29b\") " pod="openshift-ingress/e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq" Jul 13 22:35:40.045541 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:40.045446 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/911216b4-9bcb-4ef4-9a70-eba330c4a29b-istio-envoy\") pod \"e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq\" (UID: \"911216b4-9bcb-4ef4-9a70-eba330c4a29b\") " pod="openshift-ingress/e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq" Jul 13 22:35:40.045541 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:40.045473 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/911216b4-9bcb-4ef4-9a70-eba330c4a29b-istio-token\") pod \"e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq\" (UID: \"911216b4-9bcb-4ef4-9a70-eba330c4a29b\") " pod="openshift-ingress/e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq" Jul 13 22:35:40.045541 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:40.045511 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/911216b4-9bcb-4ef4-9a70-eba330c4a29b-istio-data\") pod \"e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq\" (UID: \"911216b4-9bcb-4ef4-9a70-eba330c4a29b\") " pod="openshift-ingress/e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq" Jul 13 22:35:40.045910 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:40.045548 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/911216b4-9bcb-4ef4-9a70-eba330c4a29b-workload-socket\") pod \"e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq\" (UID: \"911216b4-9bcb-4ef4-9a70-eba330c4a29b\") " pod="openshift-ingress/e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq" Jul 13 22:35:40.045910 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:40.045595 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/911216b4-9bcb-4ef4-9a70-eba330c4a29b-istiod-ca-cert\") pod \"e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq\" (UID: \"911216b4-9bcb-4ef4-9a70-eba330c4a29b\") " pod="openshift-ingress/e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq" Jul 13 22:35:40.045910 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:40.045619 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7bhw\" (UniqueName: \"kubernetes.io/projected/911216b4-9bcb-4ef4-9a70-eba330c4a29b-kube-api-access-j7bhw\") pod \"e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq\" (UID: \"911216b4-9bcb-4ef4-9a70-eba330c4a29b\") " pod="openshift-ingress/e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq" Jul 13 22:35:40.045910 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:40.045837 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/911216b4-9bcb-4ef4-9a70-eba330c4a29b-workload-certs\") pod \"e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq\" (UID: \"911216b4-9bcb-4ef4-9a70-eba330c4a29b\") " pod="openshift-ingress/e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq" Jul 13 22:35:40.045910 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:40.045889 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/911216b4-9bcb-4ef4-9a70-eba330c4a29b-istio-data\") pod \"e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq\" (UID: \"911216b4-9bcb-4ef4-9a70-eba330c4a29b\") " pod="openshift-ingress/e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq" Jul 13 22:35:40.046130 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:40.045957 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/911216b4-9bcb-4ef4-9a70-eba330c4a29b-workload-socket\") pod \"e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq\" (UID: \"911216b4-9bcb-4ef4-9a70-eba330c4a29b\") " pod="openshift-ingress/e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq" Jul 13 22:35:40.046350 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:40.046255 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/911216b4-9bcb-4ef4-9a70-eba330c4a29b-credential-socket\") pod \"e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq\" (UID: \"911216b4-9bcb-4ef4-9a70-eba330c4a29b\") " pod="openshift-ingress/e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq" Jul 13 22:35:40.046594 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:40.046564 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/911216b4-9bcb-4ef4-9a70-eba330c4a29b-istiod-ca-cert\") pod \"e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq\" (UID: \"911216b4-9bcb-4ef4-9a70-eba330c4a29b\") " pod="openshift-ingress/e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq" Jul 13 22:35:40.048035 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:40.048007 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/911216b4-9bcb-4ef4-9a70-eba330c4a29b-istio-envoy\") pod \"e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq\" (UID: \"911216b4-9bcb-4ef4-9a70-eba330c4a29b\") " pod="openshift-ingress/e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq" Jul 13 22:35:40.048239 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:40.048218 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/911216b4-9bcb-4ef4-9a70-eba330c4a29b-istio-podinfo\") pod \"e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq\" (UID: \"911216b4-9bcb-4ef4-9a70-eba330c4a29b\") " pod="openshift-ingress/e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq" Jul 13 22:35:40.055906 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:40.055887 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/911216b4-9bcb-4ef4-9a70-eba330c4a29b-istio-token\") pod \"e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq\" (UID: \"911216b4-9bcb-4ef4-9a70-eba330c4a29b\") " pod="openshift-ingress/e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq" Jul 13 22:35:40.059448 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:40.059426 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7bhw\" (UniqueName: \"kubernetes.io/projected/911216b4-9bcb-4ef4-9a70-eba330c4a29b-kube-api-access-j7bhw\") pod \"e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq\" (UID: \"911216b4-9bcb-4ef4-9a70-eba330c4a29b\") " pod="openshift-ingress/e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq" Jul 13 22:35:40.190334 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:40.190232 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq" Jul 13 22:35:40.340712 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:40.340682 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq"] Jul 13 22:35:40.341639 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:35:40.341612 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod911216b4_9bcb_4ef4_9a70_eba330c4a29b.slice/crio-695a20e5fe6381677df7d2c3a9ca6248ddc0535475c75cb36eecc001376a0e8b WatchSource:0}: Error finding container 695a20e5fe6381677df7d2c3a9ca6248ddc0535475c75cb36eecc001376a0e8b: Status 404 returned error can't find the container with id 695a20e5fe6381677df7d2c3a9ca6248ddc0535475c75cb36eecc001376a0e8b Jul 13 22:35:40.343745 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:40.343715 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:35:40.343815 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:40.343786 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:35:40.343856 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:40.343816 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:35:41.058969 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:41.058925 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq" event={"ID":"911216b4-9bcb-4ef4-9a70-eba330c4a29b","Type":"ContainerStarted","Data":"e64a9ac826a3031f5d52ff51e59a448d8f6e335545136e9525362f067a850739"} Jul 13 22:35:41.059387 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:41.059032 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq" event={"ID":"911216b4-9bcb-4ef4-9a70-eba330c4a29b","Type":"ContainerStarted","Data":"695a20e5fe6381677df7d2c3a9ca6248ddc0535475c75cb36eecc001376a0e8b"} Jul 13 22:35:41.080865 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:41.080815 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq" podStartSLOduration=2.080799385 podStartE2EDuration="2.080799385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:35:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:35:41.078689132 +0000 UTC m=+2147.072443438" watchObservedRunningTime="2026-07-13 22:35:41.080799385 +0000 UTC m=+2147.074553678" Jul 13 22:35:41.190908 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:41.190870 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq" Jul 13 22:35:41.200327 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:41.200289 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq" Jul 13 22:35:42.064821 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:42.064784 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq" Jul 13 22:35:42.065861 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:42.065841 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq" Jul 13 22:35:48.061958 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:48.061878 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf"] Jul 13 22:35:48.062585 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:48.062567 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="17eb4d3d-212c-4faf-87cf-960879e5c333" containerName="istio-proxy" Jul 13 22:35:48.066043 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:48.066025 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf" Jul 13 22:35:48.068894 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:48.068868 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-a502db-openshift-default-dockercfg-ln47l\"" Jul 13 22:35:48.076913 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:48.076887 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf"] Jul 13 22:35:48.218636 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:48.218587 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mnjq\" (UniqueName: \"kubernetes.io/projected/a3fd3512-2c06-47a2-a4f8-2d8b8347338c-kube-api-access-9mnjq\") pod \"e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf\" (UID: \"a3fd3512-2c06-47a2-a4f8-2d8b8347338c\") " pod="openshift-ingress/e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf" Jul 13 22:35:48.218849 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:48.218650 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3fd3512-2c06-47a2-a4f8-2d8b8347338c-workload-certs\") pod \"e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf\" (UID: \"a3fd3512-2c06-47a2-a4f8-2d8b8347338c\") " pod="openshift-ingress/e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf" Jul 13 22:35:48.218849 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:48.218686 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3fd3512-2c06-47a2-a4f8-2d8b8347338c-workload-socket\") pod \"e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf\" (UID: \"a3fd3512-2c06-47a2-a4f8-2d8b8347338c\") " pod="openshift-ingress/e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf" Jul 13 22:35:48.218849 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:48.218749 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3fd3512-2c06-47a2-a4f8-2d8b8347338c-istio-token\") pod \"e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf\" (UID: \"a3fd3512-2c06-47a2-a4f8-2d8b8347338c\") " pod="openshift-ingress/e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf" Jul 13 22:35:48.218849 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:48.218774 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3fd3512-2c06-47a2-a4f8-2d8b8347338c-istio-envoy\") pod \"e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf\" (UID: \"a3fd3512-2c06-47a2-a4f8-2d8b8347338c\") " pod="openshift-ingress/e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf" Jul 13 22:35:48.218849 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:48.218796 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3fd3512-2c06-47a2-a4f8-2d8b8347338c-istio-podinfo\") pod \"e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf\" (UID: \"a3fd3512-2c06-47a2-a4f8-2d8b8347338c\") " pod="openshift-ingress/e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf" Jul 13 22:35:48.218849 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:48.218838 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3fd3512-2c06-47a2-a4f8-2d8b8347338c-istio-data\") pod \"e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf\" (UID: \"a3fd3512-2c06-47a2-a4f8-2d8b8347338c\") " pod="openshift-ingress/e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf" Jul 13 22:35:48.219090 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:48.218936 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3fd3512-2c06-47a2-a4f8-2d8b8347338c-credential-socket\") pod \"e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf\" (UID: \"a3fd3512-2c06-47a2-a4f8-2d8b8347338c\") " pod="openshift-ingress/e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf" Jul 13 22:35:48.219090 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:48.218972 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3fd3512-2c06-47a2-a4f8-2d8b8347338c-istiod-ca-cert\") pod \"e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf\" (UID: \"a3fd3512-2c06-47a2-a4f8-2d8b8347338c\") " pod="openshift-ingress/e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf" Jul 13 22:35:48.320451 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:48.320347 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3fd3512-2c06-47a2-a4f8-2d8b8347338c-istio-envoy\") pod \"e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf\" (UID: \"a3fd3512-2c06-47a2-a4f8-2d8b8347338c\") " pod="openshift-ingress/e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf" Jul 13 22:35:48.320451 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:48.320388 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3fd3512-2c06-47a2-a4f8-2d8b8347338c-istio-podinfo\") pod \"e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf\" (UID: \"a3fd3512-2c06-47a2-a4f8-2d8b8347338c\") " pod="openshift-ingress/e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf" Jul 13 22:35:48.320451 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:48.320420 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3fd3512-2c06-47a2-a4f8-2d8b8347338c-istio-data\") pod \"e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf\" (UID: \"a3fd3512-2c06-47a2-a4f8-2d8b8347338c\") " pod="openshift-ingress/e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf" Jul 13 22:35:48.320731 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:48.320462 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3fd3512-2c06-47a2-a4f8-2d8b8347338c-credential-socket\") pod \"e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf\" (UID: \"a3fd3512-2c06-47a2-a4f8-2d8b8347338c\") " pod="openshift-ingress/e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf" Jul 13 22:35:48.320731 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:48.320489 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3fd3512-2c06-47a2-a4f8-2d8b8347338c-istiod-ca-cert\") pod \"e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf\" (UID: \"a3fd3512-2c06-47a2-a4f8-2d8b8347338c\") " pod="openshift-ingress/e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf" Jul 13 22:35:48.320731 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:48.320549 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mnjq\" (UniqueName: \"kubernetes.io/projected/a3fd3512-2c06-47a2-a4f8-2d8b8347338c-kube-api-access-9mnjq\") pod \"e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf\" (UID: \"a3fd3512-2c06-47a2-a4f8-2d8b8347338c\") " pod="openshift-ingress/e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf" Jul 13 22:35:48.320731 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:48.320576 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3fd3512-2c06-47a2-a4f8-2d8b8347338c-workload-certs\") pod \"e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf\" (UID: \"a3fd3512-2c06-47a2-a4f8-2d8b8347338c\") " pod="openshift-ingress/e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf" Jul 13 22:35:48.320731 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:48.320604 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3fd3512-2c06-47a2-a4f8-2d8b8347338c-workload-socket\") pod \"e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf\" (UID: \"a3fd3512-2c06-47a2-a4f8-2d8b8347338c\") " pod="openshift-ingress/e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf" Jul 13 22:35:48.320731 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:48.320650 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3fd3512-2c06-47a2-a4f8-2d8b8347338c-istio-token\") pod \"e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf\" (UID: \"a3fd3512-2c06-47a2-a4f8-2d8b8347338c\") " pod="openshift-ingress/e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf" Jul 13 22:35:48.321052 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:48.320900 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3fd3512-2c06-47a2-a4f8-2d8b8347338c-istio-data\") pod \"e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf\" (UID: \"a3fd3512-2c06-47a2-a4f8-2d8b8347338c\") " pod="openshift-ingress/e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf" Jul 13 22:35:48.321256 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:48.321062 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3fd3512-2c06-47a2-a4f8-2d8b8347338c-workload-socket\") pod \"e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf\" (UID: \"a3fd3512-2c06-47a2-a4f8-2d8b8347338c\") " pod="openshift-ingress/e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf" Jul 13 22:35:48.321256 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:48.321185 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3fd3512-2c06-47a2-a4f8-2d8b8347338c-workload-certs\") pod \"e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf\" (UID: \"a3fd3512-2c06-47a2-a4f8-2d8b8347338c\") " pod="openshift-ingress/e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf" Jul 13 22:35:48.321423 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:48.321361 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3fd3512-2c06-47a2-a4f8-2d8b8347338c-istiod-ca-cert\") pod \"e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf\" (UID: \"a3fd3512-2c06-47a2-a4f8-2d8b8347338c\") " pod="openshift-ingress/e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf" Jul 13 22:35:48.321485 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:48.321464 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3fd3512-2c06-47a2-a4f8-2d8b8347338c-credential-socket\") pod \"e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf\" (UID: \"a3fd3512-2c06-47a2-a4f8-2d8b8347338c\") " pod="openshift-ingress/e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf" Jul 13 22:35:48.322944 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:48.322914 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3fd3512-2c06-47a2-a4f8-2d8b8347338c-istio-envoy\") pod \"e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf\" (UID: \"a3fd3512-2c06-47a2-a4f8-2d8b8347338c\") " pod="openshift-ingress/e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf" Jul 13 22:35:48.323067 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:48.322992 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3fd3512-2c06-47a2-a4f8-2d8b8347338c-istio-podinfo\") pod \"e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf\" (UID: \"a3fd3512-2c06-47a2-a4f8-2d8b8347338c\") " pod="openshift-ingress/e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf" Jul 13 22:35:48.332859 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:48.332831 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3fd3512-2c06-47a2-a4f8-2d8b8347338c-istio-token\") pod \"e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf\" (UID: \"a3fd3512-2c06-47a2-a4f8-2d8b8347338c\") " pod="openshift-ingress/e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf" Jul 13 22:35:48.340110 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:48.340086 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mnjq\" (UniqueName: \"kubernetes.io/projected/a3fd3512-2c06-47a2-a4f8-2d8b8347338c-kube-api-access-9mnjq\") pod \"e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf\" (UID: \"a3fd3512-2c06-47a2-a4f8-2d8b8347338c\") " pod="openshift-ingress/e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf" Jul 13 22:35:48.378926 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:48.378895 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf" Jul 13 22:35:48.525031 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:48.525003 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf"] Jul 13 22:35:48.526232 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:35:48.526191 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3fd3512_2c06_47a2_a4f8_2d8b8347338c.slice/crio-c85818fffc44442d06e28859f939fabacb1101e32370efb7aa31178d74905902 WatchSource:0}: Error finding container c85818fffc44442d06e28859f939fabacb1101e32370efb7aa31178d74905902: Status 404 returned error can't find the container with id c85818fffc44442d06e28859f939fabacb1101e32370efb7aa31178d74905902 Jul 13 22:35:48.528433 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:48.528399 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:35:48.528506 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:48.528464 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:35:48.528506 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:48.528489 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:35:49.118667 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:49.118634 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf" event={"ID":"a3fd3512-2c06-47a2-a4f8-2d8b8347338c","Type":"ContainerStarted","Data":"bc8bdaa19f97287dceed26ce04fa84711b853dc04c8315e2dd0a93a15e490a89"} Jul 13 22:35:49.119083 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:49.118687 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf" event={"ID":"a3fd3512-2c06-47a2-a4f8-2d8b8347338c","Type":"ContainerStarted","Data":"c85818fffc44442d06e28859f939fabacb1101e32370efb7aa31178d74905902"} Jul 13 22:35:49.137694 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:49.137640 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf" podStartSLOduration=1.137621224 podStartE2EDuration="1.137621224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:35:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:35:49.137274913 +0000 UTC m=+2155.131029227" watchObservedRunningTime="2026-07-13 22:35:49.137621224 +0000 UTC m=+2155.131375518" Jul 13 22:35:49.379561 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:49.379529 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf" Jul 13 22:35:49.384522 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:49.384495 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf" Jul 13 22:35:50.124272 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:50.124238 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf" Jul 13 22:35:50.125148 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:50.125130 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf" Jul 13 22:35:58.106395 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:58.106353 2555 patch_prober.go:28] interesting pod/e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 13 22:35:58.106965 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:58.106898 2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq" podUID="911216b4-9bcb-4ef4-9a70-eba330c4a29b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 13 22:35:59.181705 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:59.181619 2555 generic.go:356] "Generic (PLEG): container finished" podID="911216b4-9bcb-4ef4-9a70-eba330c4a29b" containerID="e64a9ac826a3031f5d52ff51e59a448d8f6e335545136e9525362f067a850739" exitCode=0 Jul 13 22:35:59.181705 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:59.181693 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq" event={"ID":"911216b4-9bcb-4ef4-9a70-eba330c4a29b","Type":"ContainerDied","Data":"e64a9ac826a3031f5d52ff51e59a448d8f6e335545136e9525362f067a850739"} Jul 13 22:35:59.182224 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:59.182095 2555 scope.go:117] "RemoveContainer" containerID="e64a9ac826a3031f5d52ff51e59a448d8f6e335545136e9525362f067a850739" Jul 13 22:35:59.183118 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:59.183091 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:35:59.183277 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:59.183163 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:35:59.183277 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:35:59.183219 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:36:00.189224 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:36:00.189163 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq" event={"ID":"911216b4-9bcb-4ef4-9a70-eba330c4a29b","Type":"ContainerStarted","Data":"1e11ed12182a99bce68263fd248b4f578a9dd4611d9620426f3bc897220e32e4"} Jul 13 22:36:00.190676 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:36:00.190657 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq" Jul 13 22:36:00.190915 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:36:00.190894 2555 patch_prober.go:28] interesting pod/e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Jul 13 22:36:00.190969 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:36:00.190940 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq" podUID="911216b4-9bcb-4ef4-9a70-eba330c4a29b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Jul 13 22:36:01.191128 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:36:01.190656 2555 patch_prober.go:28] interesting pod/e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Jul 13 22:36:01.191128 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:36:01.190719 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq" podUID="911216b4-9bcb-4ef4-9a70-eba330c4a29b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Jul 13 22:36:02.191318 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:36:02.191273 2555 patch_prober.go:28] interesting pod/e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Jul 13 22:36:02.191837 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:36:02.191351 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq" podUID="911216b4-9bcb-4ef4-9a70-eba330c4a29b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Jul 13 22:36:03.191257 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:36:03.191170 2555 patch_prober.go:28] interesting pod/e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Jul 13 22:36:03.191523 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:36:03.191284 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq" podUID="911216b4-9bcb-4ef4-9a70-eba330c4a29b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Jul 13 22:36:04.211858 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:36:04.211817 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq" Jul 13 22:36:04.212500 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:36:04.212478 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq" Jul 13 22:36:04.220499 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:36:04.220429 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq" Jul 13 22:36:06.165394 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:36:06.165352 2555 patch_prober.go:28] interesting pod/e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 13 22:36:06.165803 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:36:06.165432 2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf" podUID="a3fd3512-2c06-47a2-a4f8-2d8b8347338c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 13 22:36:07.245893 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:36:07.245859 2555 generic.go:356] "Generic (PLEG): container finished" podID="a3fd3512-2c06-47a2-a4f8-2d8b8347338c" containerID="bc8bdaa19f97287dceed26ce04fa84711b853dc04c8315e2dd0a93a15e490a89" exitCode=0 Jul 13 22:36:07.246332 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:36:07.245930 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf" event={"ID":"a3fd3512-2c06-47a2-a4f8-2d8b8347338c","Type":"ContainerDied","Data":"bc8bdaa19f97287dceed26ce04fa84711b853dc04c8315e2dd0a93a15e490a89"} Jul 13 22:36:07.246377 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:36:07.246356 2555 scope.go:117] "RemoveContainer" containerID="bc8bdaa19f97287dceed26ce04fa84711b853dc04c8315e2dd0a93a15e490a89" Jul 13 22:36:07.247503 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:36:07.247474 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:36:07.247594 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:36:07.247544 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:36:07.247594 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:36:07.247575 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:36:08.253325 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:36:08.253292 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf" event={"ID":"a3fd3512-2c06-47a2-a4f8-2d8b8347338c","Type":"ContainerStarted","Data":"3212db8cbb6280b963c09c2f2873bbce479bcee6613b793bfe40346ec8438254"} Jul 13 22:36:08.379677 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:36:08.379641 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf" Jul 13 22:36:08.379879 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:36:08.379857 2555 patch_prober.go:28] interesting pod/e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Jul 13 22:36:08.379924 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:36:08.379908 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf" podUID="a3fd3512-2c06-47a2-a4f8-2d8b8347338c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Jul 13 22:36:09.380157 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:36:09.380117 2555 patch_prober.go:28] interesting pod/e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Jul 13 22:36:09.380544 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:36:09.380175 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf" podUID="a3fd3512-2c06-47a2-a4f8-2d8b8347338c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Jul 13 22:36:10.380149 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:36:10.380107 2555 patch_prober.go:28] interesting pod/e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Jul 13 22:36:10.380379 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:36:10.380168 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf" podUID="a3fd3512-2c06-47a2-a4f8-2d8b8347338c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Jul 13 22:36:11.380020 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:36:11.379983 2555 patch_prober.go:28] interesting pod/e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Jul 13 22:36:11.380220 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:36:11.380039 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf" podUID="a3fd3512-2c06-47a2-a4f8-2d8b8347338c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Jul 13 22:36:12.382421 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:36:12.382392 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf" Jul 13 22:36:12.382833 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:36:12.382724 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf" Jul 13 22:36:12.383305 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:36:12.383287 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf" Jul 13 22:36:49.999496 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:36:49.999447 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7da020-pn5mq"] Jul 13 22:36:50.005560 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:36:50.005532 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7da020-pn5mq" Jul 13 22:36:50.009421 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:36:50.009393 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 13 22:36:50.009696 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:36:50.009673 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k5z5d\"" Jul 13 22:36:50.010368 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:36:50.010345 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7da020-pn5mq"] Jul 13 22:36:50.105764 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:36:50.105729 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/976c57b7-04ab-4c1d-9c55-5e974dbb0417-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7da020-pn5mq\" (UID: \"976c57b7-04ab-4c1d-9c55-5e974dbb0417\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7da020-pn5mq" Jul 13 22:36:50.206616 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:36:50.206578 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/976c57b7-04ab-4c1d-9c55-5e974dbb0417-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7da020-pn5mq\" (UID: \"976c57b7-04ab-4c1d-9c55-5e974dbb0417\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7da020-pn5mq" Jul 13 22:36:50.207253 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:36:50.207235 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/976c57b7-04ab-4c1d-9c55-5e974dbb0417-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7da020-pn5mq\" (UID: \"976c57b7-04ab-4c1d-9c55-5e974dbb0417\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7da020-pn5mq" Jul 13 22:36:50.319300 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:36:50.319172 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7da020-pn5mq" Jul 13 22:36:50.458821 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:36:50.458791 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7da020-pn5mq"] Jul 13 22:36:50.461023 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:36:50.460997 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod976c57b7_04ab_4c1d_9c55_5e974dbb0417.slice/crio-70459ca98b1c56a985756cd657886243ed8410596da52b2fd6f589aacfb883b5 WatchSource:0}: Error finding container 70459ca98b1c56a985756cd657886243ed8410596da52b2fd6f589aacfb883b5: Status 404 returned error can't find the container with id 70459ca98b1c56a985756cd657886243ed8410596da52b2fd6f589aacfb883b5 Jul 13 22:36:50.495389 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:36:50.495362 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7da020-pn5mq" event={"ID":"976c57b7-04ab-4c1d-9c55-5e974dbb0417","Type":"ContainerStarted","Data":"70459ca98b1c56a985756cd657886243ed8410596da52b2fd6f589aacfb883b5"} Jul 13 22:36:51.503078 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:36:51.503037 2555 generic.go:356] "Generic (PLEG): container finished" podID="976c57b7-04ab-4c1d-9c55-5e974dbb0417" containerID="6f0c3e012f1caa4596bc9ed5ec79f6f589d69aaefccc9ec6af6ca29b98a7b3b5" exitCode=0 Jul 13 22:36:51.503504 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:36:51.503093 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7da020-pn5mq" event={"ID":"976c57b7-04ab-4c1d-9c55-5e974dbb0417","Type":"ContainerDied","Data":"6f0c3e012f1caa4596bc9ed5ec79f6f589d69aaefccc9ec6af6ca29b98a7b3b5"} Jul 13 22:36:52.644357 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:36:52.644334 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7da020-pn5mq" Jul 13 22:36:52.727903 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:36:52.727872 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/976c57b7-04ab-4c1d-9c55-5e974dbb0417-maas-api-service-ca\") pod \"976c57b7-04ab-4c1d-9c55-5e974dbb0417\" (UID: \"976c57b7-04ab-4c1d-9c55-5e974dbb0417\") " Jul 13 22:36:52.728263 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:36:52.728236 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/976c57b7-04ab-4c1d-9c55-5e974dbb0417-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "976c57b7-04ab-4c1d-9c55-5e974dbb0417" (UID: "976c57b7-04ab-4c1d-9c55-5e974dbb0417"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:36:52.829250 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:36:52.829148 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/976c57b7-04ab-4c1d-9c55-5e974dbb0417-maas-api-service-ca\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:36:53.515294 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:36:53.515264 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7da020-pn5mq" Jul 13 22:36:53.515294 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:36:53.515268 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7da020-pn5mq" event={"ID":"976c57b7-04ab-4c1d-9c55-5e974dbb0417","Type":"ContainerDied","Data":"70459ca98b1c56a985756cd657886243ed8410596da52b2fd6f589aacfb883b5"} Jul 13 22:36:53.515557 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:36:53.515313 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70459ca98b1c56a985756cd657886243ed8410596da52b2fd6f589aacfb883b5" Jul 13 22:37:15.749339 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:15.749249 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq"] Jul 13 22:37:15.749908 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:15.749551 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq" podUID="911216b4-9bcb-4ef4-9a70-eba330c4a29b" containerName="istio-proxy" containerID="cri-o://1e11ed12182a99bce68263fd248b4f578a9dd4611d9620426f3bc897220e32e4" gracePeriod=30 Jul 13 22:37:16.112362 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:16.112277 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a502db-gzrz2"] Jul 13 22:37:16.112669 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:16.112656 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="976c57b7-04ab-4c1d-9c55-5e974dbb0417" containerName="revoke-keys" Jul 13 22:37:16.112717 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:16.112671 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="976c57b7-04ab-4c1d-9c55-5e974dbb0417" containerName="revoke-keys" Jul 13 22:37:16.112753 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:16.112742 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="976c57b7-04ab-4c1d-9c55-5e974dbb0417" containerName="revoke-keys" Jul 13 22:37:16.115963 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:16.115944 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a502db-gzrz2" Jul 13 22:37:16.119358 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:16.119333 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k5z5d\"" Jul 13 22:37:16.121403 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:16.121381 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 13 22:37:16.139340 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:16.139319 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a502db-gzrz2"] Jul 13 22:37:16.246067 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:16.246031 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af30161a-8947-484f-b8b7-2d528d7c7d09-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a502db-gzrz2\" (UID: \"af30161a-8947-484f-b8b7-2d528d7c7d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a502db-gzrz2" Jul 13 22:37:16.346584 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:16.346547 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af30161a-8947-484f-b8b7-2d528d7c7d09-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a502db-gzrz2\" (UID: \"af30161a-8947-484f-b8b7-2d528d7c7d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a502db-gzrz2" Jul 13 22:37:16.347169 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:16.347148 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af30161a-8947-484f-b8b7-2d528d7c7d09-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a502db-gzrz2\" (UID: \"af30161a-8947-484f-b8b7-2d528d7c7d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a502db-gzrz2" Jul 13 22:37:16.425873 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:16.425841 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a502db-gzrz2" Jul 13 22:37:16.562220 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:16.562169 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a502db-gzrz2"] Jul 13 22:37:16.563897 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:37:16.563870 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf30161a_8947_484f_b8b7_2d528d7c7d09.slice/crio-1916d1db59bab569aacef2617302ff978f6aef2b6648d95cb564a521a3120af4 WatchSource:0}: Error finding container 1916d1db59bab569aacef2617302ff978f6aef2b6648d95cb564a521a3120af4: Status 404 returned error can't find the container with id 1916d1db59bab569aacef2617302ff978f6aef2b6648d95cb564a521a3120af4 Jul 13 22:37:16.651511 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:16.651430 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a502db-gzrz2" event={"ID":"af30161a-8947-484f-b8b7-2d528d7c7d09","Type":"ContainerStarted","Data":"1916d1db59bab569aacef2617302ff978f6aef2b6648d95cb564a521a3120af4"} Jul 13 22:37:17.659166 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:17.659071 2555 generic.go:356] "Generic (PLEG): container finished" podID="af30161a-8947-484f-b8b7-2d528d7c7d09" containerID="49ae746c04095eb5ecae4412c823bb3fec8e94882c0d1bf47c33a39a3a9b5f45" exitCode=0 Jul 13 22:37:17.659166 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:17.659155 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a502db-gzrz2" event={"ID":"af30161a-8947-484f-b8b7-2d528d7c7d09","Type":"ContainerDied","Data":"49ae746c04095eb5ecae4412c823bb3fec8e94882c0d1bf47c33a39a3a9b5f45"} Jul 13 22:37:18.802402 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:18.802370 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a502db-gzrz2" Jul 13 22:37:18.867764 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:18.867732 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af30161a-8947-484f-b8b7-2d528d7c7d09-maas-api-service-ca\") pod \"af30161a-8947-484f-b8b7-2d528d7c7d09\" (UID: \"af30161a-8947-484f-b8b7-2d528d7c7d09\") " Jul 13 22:37:18.868089 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:18.868067 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af30161a-8947-484f-b8b7-2d528d7c7d09-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "af30161a-8947-484f-b8b7-2d528d7c7d09" (UID: "af30161a-8947-484f-b8b7-2d528d7c7d09"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:37:18.969038 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:18.968955 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af30161a-8947-484f-b8b7-2d528d7c7d09-maas-api-service-ca\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:37:19.215377 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:19.215336 2555 patch_prober.go:28] interesting pod/e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 13 22:37:19.215539 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:19.215403 2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq" podUID="911216b4-9bcb-4ef4-9a70-eba330c4a29b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 13 22:37:19.673144 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:19.673114 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a502db-gzrz2" Jul 13 22:37:19.673337 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:19.673147 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a502db-gzrz2" event={"ID":"af30161a-8947-484f-b8b7-2d528d7c7d09","Type":"ContainerDied","Data":"1916d1db59bab569aacef2617302ff978f6aef2b6648d95cb564a521a3120af4"} Jul 13 22:37:19.673337 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:19.673193 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1916d1db59bab569aacef2617302ff978f6aef2b6648d95cb564a521a3120af4" Jul 13 22:37:21.009937 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:21.009874 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq" Jul 13 22:37:21.088015 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:21.087981 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/911216b4-9bcb-4ef4-9a70-eba330c4a29b-workload-certs\") pod \"911216b4-9bcb-4ef4-9a70-eba330c4a29b\" (UID: \"911216b4-9bcb-4ef4-9a70-eba330c4a29b\") " Jul 13 22:37:21.088189 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:21.088028 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/911216b4-9bcb-4ef4-9a70-eba330c4a29b-istio-podinfo\") pod \"911216b4-9bcb-4ef4-9a70-eba330c4a29b\" (UID: \"911216b4-9bcb-4ef4-9a70-eba330c4a29b\") " Jul 13 22:37:21.088189 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:21.088073 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/911216b4-9bcb-4ef4-9a70-eba330c4a29b-workload-socket\") pod \"911216b4-9bcb-4ef4-9a70-eba330c4a29b\" (UID: \"911216b4-9bcb-4ef4-9a70-eba330c4a29b\") " Jul 13 22:37:21.088189 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:21.088123 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/911216b4-9bcb-4ef4-9a70-eba330c4a29b-istio-envoy\") pod \"911216b4-9bcb-4ef4-9a70-eba330c4a29b\" (UID: \"911216b4-9bcb-4ef4-9a70-eba330c4a29b\") " Jul 13 22:37:21.088365 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:21.088332 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/911216b4-9bcb-4ef4-9a70-eba330c4a29b-istiod-ca-cert\") pod \"911216b4-9bcb-4ef4-9a70-eba330c4a29b\" (UID: \"911216b4-9bcb-4ef4-9a70-eba330c4a29b\") " Jul 13 22:37:21.088365 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:21.088348 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/911216b4-9bcb-4ef4-9a70-eba330c4a29b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "911216b4-9bcb-4ef4-9a70-eba330c4a29b" (UID: "911216b4-9bcb-4ef4-9a70-eba330c4a29b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:37:21.088479 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:21.088384 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j7bhw\" (UniqueName: \"kubernetes.io/projected/911216b4-9bcb-4ef4-9a70-eba330c4a29b-kube-api-access-j7bhw\") pod \"911216b4-9bcb-4ef4-9a70-eba330c4a29b\" (UID: \"911216b4-9bcb-4ef4-9a70-eba330c4a29b\") " Jul 13 22:37:21.088479 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:21.088418 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/911216b4-9bcb-4ef4-9a70-eba330c4a29b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "911216b4-9bcb-4ef4-9a70-eba330c4a29b" (UID: "911216b4-9bcb-4ef4-9a70-eba330c4a29b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:37:21.088479 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:21.088436 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/911216b4-9bcb-4ef4-9a70-eba330c4a29b-istio-data\") pod \"911216b4-9bcb-4ef4-9a70-eba330c4a29b\" (UID: \"911216b4-9bcb-4ef4-9a70-eba330c4a29b\") " Jul 13 22:37:21.088627 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:21.088504 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/911216b4-9bcb-4ef4-9a70-eba330c4a29b-istio-token\") pod \"911216b4-9bcb-4ef4-9a70-eba330c4a29b\" (UID: \"911216b4-9bcb-4ef4-9a70-eba330c4a29b\") " Jul 13 22:37:21.088627 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:21.088535 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/911216b4-9bcb-4ef4-9a70-eba330c4a29b-credential-socket\") pod \"911216b4-9bcb-4ef4-9a70-eba330c4a29b\" (UID: \"911216b4-9bcb-4ef4-9a70-eba330c4a29b\") " Jul 13 22:37:21.088767 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:21.088734 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/911216b4-9bcb-4ef4-9a70-eba330c4a29b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "911216b4-9bcb-4ef4-9a70-eba330c4a29b" (UID: "911216b4-9bcb-4ef4-9a70-eba330c4a29b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:37:21.088860 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:21.088843 2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/911216b4-9bcb-4ef4-9a70-eba330c4a29b-workload-certs\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:37:21.088918 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:21.088878 2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/911216b4-9bcb-4ef4-9a70-eba330c4a29b-workload-socket\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:37:21.088918 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:21.088895 2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/911216b4-9bcb-4ef4-9a70-eba330c4a29b-istiod-ca-cert\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:37:21.089094 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:21.089073 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/911216b4-9bcb-4ef4-9a70-eba330c4a29b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "911216b4-9bcb-4ef4-9a70-eba330c4a29b" (UID: "911216b4-9bcb-4ef4-9a70-eba330c4a29b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:37:21.089631 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:21.089601 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/911216b4-9bcb-4ef4-9a70-eba330c4a29b-istio-data" (OuterVolumeSpecName: "istio-data") pod "911216b4-9bcb-4ef4-9a70-eba330c4a29b" (UID: "911216b4-9bcb-4ef4-9a70-eba330c4a29b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:37:21.090607 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:21.090575 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/911216b4-9bcb-4ef4-9a70-eba330c4a29b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "911216b4-9bcb-4ef4-9a70-eba330c4a29b" (UID: "911216b4-9bcb-4ef4-9a70-eba330c4a29b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:37:21.090721 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:21.090633 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/911216b4-9bcb-4ef4-9a70-eba330c4a29b-kube-api-access-j7bhw" (OuterVolumeSpecName: "kube-api-access-j7bhw") pod "911216b4-9bcb-4ef4-9a70-eba330c4a29b" (UID: "911216b4-9bcb-4ef4-9a70-eba330c4a29b"). InnerVolumeSpecName "kube-api-access-j7bhw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:37:21.090721 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:21.090655 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/911216b4-9bcb-4ef4-9a70-eba330c4a29b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "911216b4-9bcb-4ef4-9a70-eba330c4a29b" (UID: "911216b4-9bcb-4ef4-9a70-eba330c4a29b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 13 22:37:21.090852 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:21.090830 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/911216b4-9bcb-4ef4-9a70-eba330c4a29b-istio-token" (OuterVolumeSpecName: "istio-token") pod "911216b4-9bcb-4ef4-9a70-eba330c4a29b" (UID: "911216b4-9bcb-4ef4-9a70-eba330c4a29b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:37:21.189501 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:21.189466 2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/911216b4-9bcb-4ef4-9a70-eba330c4a29b-istio-token\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:37:21.189501 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:21.189496 2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/911216b4-9bcb-4ef4-9a70-eba330c4a29b-credential-socket\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:37:21.189501 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:21.189504 2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/911216b4-9bcb-4ef4-9a70-eba330c4a29b-istio-podinfo\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:37:21.189501 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:21.189512 2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/911216b4-9bcb-4ef4-9a70-eba330c4a29b-istio-envoy\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:37:21.189765 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:21.189522 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7bhw\" (UniqueName: \"kubernetes.io/projected/911216b4-9bcb-4ef4-9a70-eba330c4a29b-kube-api-access-j7bhw\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:37:21.189765 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:21.189530 2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/911216b4-9bcb-4ef4-9a70-eba330c4a29b-istio-data\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:37:21.685342 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:21.685311 2555 generic.go:356] "Generic (PLEG): container finished" podID="911216b4-9bcb-4ef4-9a70-eba330c4a29b" containerID="1e11ed12182a99bce68263fd248b4f578a9dd4611d9620426f3bc897220e32e4" exitCode=0 Jul 13 22:37:21.685499 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:21.685362 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq" Jul 13 22:37:21.685499 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:21.685367 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq" event={"ID":"911216b4-9bcb-4ef4-9a70-eba330c4a29b","Type":"ContainerDied","Data":"1e11ed12182a99bce68263fd248b4f578a9dd4611d9620426f3bc897220e32e4"} Jul 13 22:37:21.685499 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:21.685478 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq" event={"ID":"911216b4-9bcb-4ef4-9a70-eba330c4a29b","Type":"ContainerDied","Data":"695a20e5fe6381677df7d2c3a9ca6248ddc0535475c75cb36eecc001376a0e8b"} Jul 13 22:37:21.685627 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:21.685503 2555 scope.go:117] "RemoveContainer" containerID="1e11ed12182a99bce68263fd248b4f578a9dd4611d9620426f3bc897220e32e4" Jul 13 22:37:21.697221 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:21.697181 2555 scope.go:117] "RemoveContainer" containerID="e64a9ac826a3031f5d52ff51e59a448d8f6e335545136e9525362f067a850739" Jul 13 22:37:21.707578 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:21.707557 2555 scope.go:117] "RemoveContainer" containerID="1e11ed12182a99bce68263fd248b4f578a9dd4611d9620426f3bc897220e32e4" Jul 13 22:37:21.707866 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:37:21.707840 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e11ed12182a99bce68263fd248b4f578a9dd4611d9620426f3bc897220e32e4\": container with ID starting with 1e11ed12182a99bce68263fd248b4f578a9dd4611d9620426f3bc897220e32e4 not found: ID does not exist" containerID="1e11ed12182a99bce68263fd248b4f578a9dd4611d9620426f3bc897220e32e4" Jul 13 22:37:21.707929 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:21.707882 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e11ed12182a99bce68263fd248b4f578a9dd4611d9620426f3bc897220e32e4"} err="failed to get container status \"1e11ed12182a99bce68263fd248b4f578a9dd4611d9620426f3bc897220e32e4\": rpc error: code = NotFound desc = could not find container \"1e11ed12182a99bce68263fd248b4f578a9dd4611d9620426f3bc897220e32e4\": container with ID starting with 1e11ed12182a99bce68263fd248b4f578a9dd4611d9620426f3bc897220e32e4 not found: ID does not exist" Jul 13 22:37:21.707929 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:21.707908 2555 scope.go:117] "RemoveContainer" containerID="e64a9ac826a3031f5d52ff51e59a448d8f6e335545136e9525362f067a850739" Jul 13 22:37:21.708263 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:37:21.708181 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e64a9ac826a3031f5d52ff51e59a448d8f6e335545136e9525362f067a850739\": container with ID starting with e64a9ac826a3031f5d52ff51e59a448d8f6e335545136e9525362f067a850739 not found: ID does not exist" containerID="e64a9ac826a3031f5d52ff51e59a448d8f6e335545136e9525362f067a850739" Jul 13 22:37:21.708263 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:21.708223 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e64a9ac826a3031f5d52ff51e59a448d8f6e335545136e9525362f067a850739"} err="failed to get container status \"e64a9ac826a3031f5d52ff51e59a448d8f6e335545136e9525362f067a850739\": rpc error: code = NotFound desc = could not find container \"e64a9ac826a3031f5d52ff51e59a448d8f6e335545136e9525362f067a850739\": container with ID starting with e64a9ac826a3031f5d52ff51e59a448d8f6e335545136e9525362f067a850739 not found: ID does not exist" Jul 13 22:37:21.710792 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:21.710757 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq"] Jul 13 22:37:21.713489 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:21.713468 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-7da020-openshift-default-6d989854d9-dk7xq"] Jul 13 22:37:22.559367 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:22.559335 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="911216b4-9bcb-4ef4-9a70-eba330c4a29b" path="/var/lib/kubelet/pods/911216b4-9bcb-4ef4-9a70-eba330c4a29b/volumes" Jul 13 22:37:41.854717 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:41.854682 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf"] Jul 13 22:37:41.855110 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:41.854914 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf" podUID="a3fd3512-2c06-47a2-a4f8-2d8b8347338c" containerName="istio-proxy" containerID="cri-o://3212db8cbb6280b963c09c2f2873bbce479bcee6613b793bfe40346ec8438254" gracePeriod=30 Jul 13 22:37:42.384875 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:42.384840 2555 patch_prober.go:28] interesting pod/e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 13 22:37:42.385046 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:42.384895 2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf" podUID="a3fd3512-2c06-47a2-a4f8-2d8b8347338c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 13 22:37:42.794751 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:42.794653 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5"] Jul 13 22:37:42.795432 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:42.795404 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af30161a-8947-484f-b8b7-2d528d7c7d09" containerName="revoke-keys" Jul 13 22:37:42.795432 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:42.795432 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="af30161a-8947-484f-b8b7-2d528d7c7d09" containerName="revoke-keys" Jul 13 22:37:42.795652 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:42.795448 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="911216b4-9bcb-4ef4-9a70-eba330c4a29b" containerName="istio-proxy" Jul 13 22:37:42.795652 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:42.795456 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="911216b4-9bcb-4ef4-9a70-eba330c4a29b" containerName="istio-proxy" Jul 13 22:37:42.795652 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:42.795467 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="911216b4-9bcb-4ef4-9a70-eba330c4a29b" containerName="istio-proxy" Jul 13 22:37:42.795652 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:42.795475 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="911216b4-9bcb-4ef4-9a70-eba330c4a29b" containerName="istio-proxy" Jul 13 22:37:42.795652 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:42.795572 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="911216b4-9bcb-4ef4-9a70-eba330c4a29b" containerName="istio-proxy" Jul 13 22:37:42.795652 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:42.795584 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="911216b4-9bcb-4ef4-9a70-eba330c4a29b" containerName="istio-proxy" Jul 13 22:37:42.795652 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:42.795594 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="af30161a-8947-484f-b8b7-2d528d7c7d09" containerName="revoke-keys" Jul 13 22:37:42.798745 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:42.798723 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5" Jul 13 22:37:42.802374 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:42.802352 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-fe823c-openshift-default-dockercfg-9hxc9\"" Jul 13 22:37:42.825544 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:42.825510 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5"] Jul 13 22:37:42.872535 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:42.872498 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/135404e9-ef6b-4702-8fd6-2642fee452f2-workload-certs\") pod \"e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5\" (UID: \"135404e9-ef6b-4702-8fd6-2642fee452f2\") " pod="openshift-ingress/e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5" Jul 13 22:37:42.872981 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:42.872576 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/135404e9-ef6b-4702-8fd6-2642fee452f2-istio-podinfo\") pod \"e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5\" (UID: \"135404e9-ef6b-4702-8fd6-2642fee452f2\") " pod="openshift-ingress/e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5" Jul 13 22:37:42.872981 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:42.872600 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d9nr\" (UniqueName: \"kubernetes.io/projected/135404e9-ef6b-4702-8fd6-2642fee452f2-kube-api-access-5d9nr\") pod \"e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5\" (UID: \"135404e9-ef6b-4702-8fd6-2642fee452f2\") " pod="openshift-ingress/e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5" Jul 13 22:37:42.872981 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:42.872620 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/135404e9-ef6b-4702-8fd6-2642fee452f2-workload-socket\") pod \"e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5\" (UID: \"135404e9-ef6b-4702-8fd6-2642fee452f2\") " pod="openshift-ingress/e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5" Jul 13 22:37:42.872981 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:42.872641 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/135404e9-ef6b-4702-8fd6-2642fee452f2-istiod-ca-cert\") pod \"e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5\" (UID: \"135404e9-ef6b-4702-8fd6-2642fee452f2\") " pod="openshift-ingress/e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5" Jul 13 22:37:42.872981 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:42.872762 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/135404e9-ef6b-4702-8fd6-2642fee452f2-istio-data\") pod \"e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5\" (UID: \"135404e9-ef6b-4702-8fd6-2642fee452f2\") " pod="openshift-ingress/e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5" Jul 13 22:37:42.872981 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:42.872821 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/135404e9-ef6b-4702-8fd6-2642fee452f2-istio-token\") pod \"e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5\" (UID: \"135404e9-ef6b-4702-8fd6-2642fee452f2\") " pod="openshift-ingress/e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5" Jul 13 22:37:42.872981 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:42.872857 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/135404e9-ef6b-4702-8fd6-2642fee452f2-credential-socket\") pod \"e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5\" (UID: \"135404e9-ef6b-4702-8fd6-2642fee452f2\") " pod="openshift-ingress/e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5" Jul 13 22:37:42.872981 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:42.872880 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/135404e9-ef6b-4702-8fd6-2642fee452f2-istio-envoy\") pod \"e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5\" (UID: \"135404e9-ef6b-4702-8fd6-2642fee452f2\") " pod="openshift-ingress/e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5" Jul 13 22:37:42.973346 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:42.973306 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/135404e9-ef6b-4702-8fd6-2642fee452f2-istio-data\") pod \"e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5\" (UID: \"135404e9-ef6b-4702-8fd6-2642fee452f2\") " pod="openshift-ingress/e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5" Jul 13 22:37:42.973516 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:42.973363 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/135404e9-ef6b-4702-8fd6-2642fee452f2-istio-token\") pod \"e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5\" (UID: \"135404e9-ef6b-4702-8fd6-2642fee452f2\") " pod="openshift-ingress/e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5" Jul 13 22:37:42.973516 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:42.973402 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/135404e9-ef6b-4702-8fd6-2642fee452f2-credential-socket\") pod \"e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5\" (UID: \"135404e9-ef6b-4702-8fd6-2642fee452f2\") " pod="openshift-ingress/e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5" Jul 13 22:37:42.973516 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:42.973423 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/135404e9-ef6b-4702-8fd6-2642fee452f2-istio-envoy\") pod \"e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5\" (UID: \"135404e9-ef6b-4702-8fd6-2642fee452f2\") " pod="openshift-ingress/e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5" Jul 13 22:37:42.973516 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:42.973462 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/135404e9-ef6b-4702-8fd6-2642fee452f2-workload-certs\") pod \"e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5\" (UID: \"135404e9-ef6b-4702-8fd6-2642fee452f2\") " pod="openshift-ingress/e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5" Jul 13 22:37:42.973749 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:42.973531 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/135404e9-ef6b-4702-8fd6-2642fee452f2-istio-podinfo\") pod \"e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5\" (UID: \"135404e9-ef6b-4702-8fd6-2642fee452f2\") " pod="openshift-ingress/e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5" Jul 13 22:37:42.973749 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:42.973559 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d9nr\" (UniqueName: \"kubernetes.io/projected/135404e9-ef6b-4702-8fd6-2642fee452f2-kube-api-access-5d9nr\") pod \"e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5\" (UID: \"135404e9-ef6b-4702-8fd6-2642fee452f2\") " pod="openshift-ingress/e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5" Jul 13 22:37:42.973749 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:42.973590 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/135404e9-ef6b-4702-8fd6-2642fee452f2-workload-socket\") pod \"e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5\" (UID: \"135404e9-ef6b-4702-8fd6-2642fee452f2\") " pod="openshift-ingress/e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5" Jul 13 22:37:42.973749 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:42.973614 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/135404e9-ef6b-4702-8fd6-2642fee452f2-istiod-ca-cert\") pod \"e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5\" (UID: \"135404e9-ef6b-4702-8fd6-2642fee452f2\") " pod="openshift-ingress/e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5" Jul 13 22:37:42.973949 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:42.973919 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/135404e9-ef6b-4702-8fd6-2642fee452f2-workload-certs\") pod \"e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5\" (UID: \"135404e9-ef6b-4702-8fd6-2642fee452f2\") " pod="openshift-ingress/e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5" Jul 13 22:37:42.974042 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:42.974020 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/135404e9-ef6b-4702-8fd6-2642fee452f2-istio-data\") pod \"e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5\" (UID: \"135404e9-ef6b-4702-8fd6-2642fee452f2\") " pod="openshift-ingress/e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5" Jul 13 22:37:42.974137 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:42.974112 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/135404e9-ef6b-4702-8fd6-2642fee452f2-credential-socket\") pod \"e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5\" (UID: \"135404e9-ef6b-4702-8fd6-2642fee452f2\") " pod="openshift-ingress/e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5" Jul 13 22:37:42.974218 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:42.974173 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/135404e9-ef6b-4702-8fd6-2642fee452f2-workload-socket\") pod \"e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5\" (UID: \"135404e9-ef6b-4702-8fd6-2642fee452f2\") " pod="openshift-ingress/e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5" Jul 13 22:37:42.974424 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:42.974404 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/135404e9-ef6b-4702-8fd6-2642fee452f2-istiod-ca-cert\") pod \"e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5\" (UID: \"135404e9-ef6b-4702-8fd6-2642fee452f2\") " pod="openshift-ingress/e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5" Jul 13 22:37:42.975911 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:42.975891 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/135404e9-ef6b-4702-8fd6-2642fee452f2-istio-envoy\") pod \"e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5\" (UID: \"135404e9-ef6b-4702-8fd6-2642fee452f2\") " pod="openshift-ingress/e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5" Jul 13 22:37:42.976155 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:42.976135 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/135404e9-ef6b-4702-8fd6-2642fee452f2-istio-podinfo\") pod \"e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5\" (UID: \"135404e9-ef6b-4702-8fd6-2642fee452f2\") " pod="openshift-ingress/e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5" Jul 13 22:37:42.981649 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:42.981623 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/135404e9-ef6b-4702-8fd6-2642fee452f2-istio-token\") pod \"e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5\" (UID: \"135404e9-ef6b-4702-8fd6-2642fee452f2\") " pod="openshift-ingress/e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5" Jul 13 22:37:42.981871 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:42.981849 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d9nr\" (UniqueName: \"kubernetes.io/projected/135404e9-ef6b-4702-8fd6-2642fee452f2-kube-api-access-5d9nr\") pod \"e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5\" (UID: \"135404e9-ef6b-4702-8fd6-2642fee452f2\") " pod="openshift-ingress/e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5" Jul 13 22:37:43.110945 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:43.110852 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5" Jul 13 22:37:43.272760 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:43.272733 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5"] Jul 13 22:37:43.273656 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:37:43.273625 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod135404e9_ef6b_4702_8fd6_2642fee452f2.slice/crio-740b030ad9c9f35520ed58666a570e91c69ff38d48341a77c2e1a730a9d4637f WatchSource:0}: Error finding container 740b030ad9c9f35520ed58666a570e91c69ff38d48341a77c2e1a730a9d4637f: Status 404 returned error can't find the container with id 740b030ad9c9f35520ed58666a570e91c69ff38d48341a77c2e1a730a9d4637f Jul 13 22:37:43.275771 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:43.275740 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:37:43.275826 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:43.275814 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:37:43.275857 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:43.275848 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:37:43.814038 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:43.814001 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5" event={"ID":"135404e9-ef6b-4702-8fd6-2642fee452f2","Type":"ContainerStarted","Data":"6fd2b6fe89b3ed6de6cb84c162f3c0fa47b5c7c4afa427cf417bc30df9141e44"} Jul 13 22:37:43.814279 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:43.814052 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5" event={"ID":"135404e9-ef6b-4702-8fd6-2642fee452f2","Type":"ContainerStarted","Data":"740b030ad9c9f35520ed58666a570e91c69ff38d48341a77c2e1a730a9d4637f"} Jul 13 22:37:43.842658 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:43.842583 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5" podStartSLOduration=1.842547089 podStartE2EDuration="1.842547089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:37:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:37:43.836147852 +0000 UTC m=+2269.829902146" watchObservedRunningTime="2026-07-13 22:37:43.842547089 +0000 UTC m=+2269.836301386" Jul 13 22:37:44.111847 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:44.111757 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5" Jul 13 22:37:44.116954 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:44.116927 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5" Jul 13 22:37:44.820336 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:44.820304 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5" Jul 13 22:37:44.821293 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:44.821273 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5" Jul 13 22:37:47.113355 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:47.113330 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf" Jul 13 22:37:47.217304 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:47.217273 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3fd3512-2c06-47a2-a4f8-2d8b8347338c-istio-token\") pod \"a3fd3512-2c06-47a2-a4f8-2d8b8347338c\" (UID: \"a3fd3512-2c06-47a2-a4f8-2d8b8347338c\") " Jul 13 22:37:47.217502 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:47.217325 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3fd3512-2c06-47a2-a4f8-2d8b8347338c-istio-data\") pod \"a3fd3512-2c06-47a2-a4f8-2d8b8347338c\" (UID: \"a3fd3512-2c06-47a2-a4f8-2d8b8347338c\") " Jul 13 22:37:47.217502 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:47.217353 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9mnjq\" (UniqueName: \"kubernetes.io/projected/a3fd3512-2c06-47a2-a4f8-2d8b8347338c-kube-api-access-9mnjq\") pod \"a3fd3512-2c06-47a2-a4f8-2d8b8347338c\" (UID: \"a3fd3512-2c06-47a2-a4f8-2d8b8347338c\") " Jul 13 22:37:47.217502 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:47.217377 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3fd3512-2c06-47a2-a4f8-2d8b8347338c-istio-envoy\") pod \"a3fd3512-2c06-47a2-a4f8-2d8b8347338c\" (UID: \"a3fd3512-2c06-47a2-a4f8-2d8b8347338c\") " Jul 13 22:37:47.217502 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:47.217402 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3fd3512-2c06-47a2-a4f8-2d8b8347338c-workload-socket\") pod \"a3fd3512-2c06-47a2-a4f8-2d8b8347338c\" (UID: \"a3fd3512-2c06-47a2-a4f8-2d8b8347338c\") " Jul 13 22:37:47.217502 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:47.217436 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3fd3512-2c06-47a2-a4f8-2d8b8347338c-istiod-ca-cert\") pod \"a3fd3512-2c06-47a2-a4f8-2d8b8347338c\" (UID: \"a3fd3512-2c06-47a2-a4f8-2d8b8347338c\") " Jul 13 22:37:47.217502 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:47.217486 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3fd3512-2c06-47a2-a4f8-2d8b8347338c-workload-certs\") pod \"a3fd3512-2c06-47a2-a4f8-2d8b8347338c\" (UID: \"a3fd3512-2c06-47a2-a4f8-2d8b8347338c\") " Jul 13 22:37:47.217848 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:47.217521 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3fd3512-2c06-47a2-a4f8-2d8b8347338c-credential-socket\") pod \"a3fd3512-2c06-47a2-a4f8-2d8b8347338c\" (UID: \"a3fd3512-2c06-47a2-a4f8-2d8b8347338c\") " Jul 13 22:37:47.217848 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:47.217574 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3fd3512-2c06-47a2-a4f8-2d8b8347338c-istio-podinfo\") pod \"a3fd3512-2c06-47a2-a4f8-2d8b8347338c\" (UID: \"a3fd3512-2c06-47a2-a4f8-2d8b8347338c\") " Jul 13 22:37:47.217848 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:47.217752 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3fd3512-2c06-47a2-a4f8-2d8b8347338c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a3fd3512-2c06-47a2-a4f8-2d8b8347338c" (UID: "a3fd3512-2c06-47a2-a4f8-2d8b8347338c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:37:47.217848 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:47.217786 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3fd3512-2c06-47a2-a4f8-2d8b8347338c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a3fd3512-2c06-47a2-a4f8-2d8b8347338c" (UID: "a3fd3512-2c06-47a2-a4f8-2d8b8347338c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:37:47.218070 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:47.217882 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3fd3512-2c06-47a2-a4f8-2d8b8347338c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a3fd3512-2c06-47a2-a4f8-2d8b8347338c" (UID: "a3fd3512-2c06-47a2-a4f8-2d8b8347338c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:37:47.218070 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:47.218005 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3fd3512-2c06-47a2-a4f8-2d8b8347338c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a3fd3512-2c06-47a2-a4f8-2d8b8347338c" (UID: "a3fd3512-2c06-47a2-a4f8-2d8b8347338c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:37:47.218070 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:47.218030 2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3fd3512-2c06-47a2-a4f8-2d8b8347338c-workload-socket\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:37:47.218070 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:47.218070 2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3fd3512-2c06-47a2-a4f8-2d8b8347338c-istiod-ca-cert\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:37:47.218070 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:47.218089 2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3fd3512-2c06-47a2-a4f8-2d8b8347338c-workload-certs\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:37:47.218922 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:47.218894 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3fd3512-2c06-47a2-a4f8-2d8b8347338c-istio-data" (OuterVolumeSpecName: "istio-data") pod "a3fd3512-2c06-47a2-a4f8-2d8b8347338c" (UID: "a3fd3512-2c06-47a2-a4f8-2d8b8347338c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:37:47.219602 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:47.219576 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3fd3512-2c06-47a2-a4f8-2d8b8347338c-istio-token" (OuterVolumeSpecName: "istio-token") pod "a3fd3512-2c06-47a2-a4f8-2d8b8347338c" (UID: "a3fd3512-2c06-47a2-a4f8-2d8b8347338c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:37:47.219697 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:47.219676 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3fd3512-2c06-47a2-a4f8-2d8b8347338c-kube-api-access-9mnjq" (OuterVolumeSpecName: "kube-api-access-9mnjq") pod "a3fd3512-2c06-47a2-a4f8-2d8b8347338c" (UID: "a3fd3512-2c06-47a2-a4f8-2d8b8347338c"). InnerVolumeSpecName "kube-api-access-9mnjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:37:47.219770 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:47.219733 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3fd3512-2c06-47a2-a4f8-2d8b8347338c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a3fd3512-2c06-47a2-a4f8-2d8b8347338c" (UID: "a3fd3512-2c06-47a2-a4f8-2d8b8347338c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:37:47.220087 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:47.220071 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3fd3512-2c06-47a2-a4f8-2d8b8347338c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a3fd3512-2c06-47a2-a4f8-2d8b8347338c" (UID: "a3fd3512-2c06-47a2-a4f8-2d8b8347338c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 13 22:37:47.319043 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:47.319008 2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3fd3512-2c06-47a2-a4f8-2d8b8347338c-credential-socket\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:37:47.319043 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:47.319038 2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3fd3512-2c06-47a2-a4f8-2d8b8347338c-istio-podinfo\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:37:47.319043 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:47.319047 2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3fd3512-2c06-47a2-a4f8-2d8b8347338c-istio-token\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:37:47.319312 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:47.319055 2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3fd3512-2c06-47a2-a4f8-2d8b8347338c-istio-data\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:37:47.319312 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:47.319065 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mnjq\" (UniqueName: \"kubernetes.io/projected/a3fd3512-2c06-47a2-a4f8-2d8b8347338c-kube-api-access-9mnjq\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:37:47.319312 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:47.319073 2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3fd3512-2c06-47a2-a4f8-2d8b8347338c-istio-envoy\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:37:47.840095 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:47.840061 2555 generic.go:356] "Generic (PLEG): container finished" podID="a3fd3512-2c06-47a2-a4f8-2d8b8347338c" containerID="3212db8cbb6280b963c09c2f2873bbce479bcee6613b793bfe40346ec8438254" exitCode=0 Jul 13 22:37:47.840295 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:47.840107 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf" Jul 13 22:37:47.840295 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:47.840149 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf" event={"ID":"a3fd3512-2c06-47a2-a4f8-2d8b8347338c","Type":"ContainerDied","Data":"3212db8cbb6280b963c09c2f2873bbce479bcee6613b793bfe40346ec8438254"} Jul 13 22:37:47.840295 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:47.840232 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf" event={"ID":"a3fd3512-2c06-47a2-a4f8-2d8b8347338c","Type":"ContainerDied","Data":"c85818fffc44442d06e28859f939fabacb1101e32370efb7aa31178d74905902"} Jul 13 22:37:47.840295 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:47.840263 2555 scope.go:117] "RemoveContainer" containerID="3212db8cbb6280b963c09c2f2873bbce479bcee6613b793bfe40346ec8438254" Jul 13 22:37:47.850069 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:47.850051 2555 scope.go:117] "RemoveContainer" containerID="bc8bdaa19f97287dceed26ce04fa84711b853dc04c8315e2dd0a93a15e490a89" Jul 13 22:37:47.858395 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:47.858379 2555 scope.go:117] "RemoveContainer" containerID="3212db8cbb6280b963c09c2f2873bbce479bcee6613b793bfe40346ec8438254" Jul 13 22:37:47.858629 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:37:47.858609 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3212db8cbb6280b963c09c2f2873bbce479bcee6613b793bfe40346ec8438254\": container with ID starting with 3212db8cbb6280b963c09c2f2873bbce479bcee6613b793bfe40346ec8438254 not found: ID does not exist" containerID="3212db8cbb6280b963c09c2f2873bbce479bcee6613b793bfe40346ec8438254" Jul 13 22:37:47.858679 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:47.858637 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3212db8cbb6280b963c09c2f2873bbce479bcee6613b793bfe40346ec8438254"} err="failed to get container status \"3212db8cbb6280b963c09c2f2873bbce479bcee6613b793bfe40346ec8438254\": rpc error: code = NotFound desc = could not find container \"3212db8cbb6280b963c09c2f2873bbce479bcee6613b793bfe40346ec8438254\": container with ID starting with 3212db8cbb6280b963c09c2f2873bbce479bcee6613b793bfe40346ec8438254 not found: ID does not exist" Jul 13 22:37:47.858679 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:47.858654 2555 scope.go:117] "RemoveContainer" containerID="bc8bdaa19f97287dceed26ce04fa84711b853dc04c8315e2dd0a93a15e490a89" Jul 13 22:37:47.858856 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:37:47.858842 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc8bdaa19f97287dceed26ce04fa84711b853dc04c8315e2dd0a93a15e490a89\": container with ID starting with bc8bdaa19f97287dceed26ce04fa84711b853dc04c8315e2dd0a93a15e490a89 not found: ID does not exist" containerID="bc8bdaa19f97287dceed26ce04fa84711b853dc04c8315e2dd0a93a15e490a89" Jul 13 22:37:47.858897 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:47.858860 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc8bdaa19f97287dceed26ce04fa84711b853dc04c8315e2dd0a93a15e490a89"} err="failed to get container status \"bc8bdaa19f97287dceed26ce04fa84711b853dc04c8315e2dd0a93a15e490a89\": rpc error: code = NotFound desc = could not find container \"bc8bdaa19f97287dceed26ce04fa84711b853dc04c8315e2dd0a93a15e490a89\": container with ID starting with bc8bdaa19f97287dceed26ce04fa84711b853dc04c8315e2dd0a93a15e490a89 not found: ID does not exist" Jul 13 22:37:47.863341 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:47.863321 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf"] Jul 13 22:37:47.874274 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:47.874251 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-a502db-openshift-default-5bd6f66987-vxdhf"] Jul 13 22:37:48.559038 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:48.559006 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3fd3512-2c06-47a2-a4f8-2d8b8347338c" path="/var/lib/kubelet/pods/a3fd3512-2c06-47a2-a4f8-2d8b8347338c/volumes" Jul 13 22:37:51.074688 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:51.074651 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb"] Jul 13 22:37:51.075274 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:51.075253 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3fd3512-2c06-47a2-a4f8-2d8b8347338c" containerName="istio-proxy" Jul 13 22:37:51.075374 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:51.075276 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3fd3512-2c06-47a2-a4f8-2d8b8347338c" containerName="istio-proxy" Jul 13 22:37:51.075374 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:51.075304 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3fd3512-2c06-47a2-a4f8-2d8b8347338c" containerName="istio-proxy" Jul 13 22:37:51.075374 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:51.075313 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3fd3512-2c06-47a2-a4f8-2d8b8347338c" containerName="istio-proxy" Jul 13 22:37:51.075531 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:51.075408 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3fd3512-2c06-47a2-a4f8-2d8b8347338c" containerName="istio-proxy" Jul 13 22:37:51.075531 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:51.075421 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3fd3512-2c06-47a2-a4f8-2d8b8347338c" containerName="istio-proxy" Jul 13 22:37:51.080216 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:51.080175 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb" Jul 13 22:37:51.083066 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:51.083047 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-2a08d3-openshift-default-dockercfg-tznjr\"" Jul 13 22:37:51.089344 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:51.089324 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb"] Jul 13 22:37:51.151483 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:51.151443 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7d032f3-6e35-41e0-9367-9fad2a4cbb57-istiod-ca-cert\") pod \"e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb\" (UID: \"c7d032f3-6e35-41e0-9367-9fad2a4cbb57\") " pod="openshift-ingress/e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb" Jul 13 22:37:51.151673 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:51.151492 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7d032f3-6e35-41e0-9367-9fad2a4cbb57-workload-certs\") pod \"e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb\" (UID: \"c7d032f3-6e35-41e0-9367-9fad2a4cbb57\") " pod="openshift-ingress/e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb" Jul 13 22:37:51.151673 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:51.151535 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7d032f3-6e35-41e0-9367-9fad2a4cbb57-istio-podinfo\") pod \"e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb\" (UID: \"c7d032f3-6e35-41e0-9367-9fad2a4cbb57\") " pod="openshift-ingress/e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb" Jul 13 22:37:51.151673 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:51.151551 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7d032f3-6e35-41e0-9367-9fad2a4cbb57-credential-socket\") pod \"e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb\" (UID: \"c7d032f3-6e35-41e0-9367-9fad2a4cbb57\") " pod="openshift-ingress/e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb" Jul 13 22:37:51.151673 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:51.151583 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7d032f3-6e35-41e0-9367-9fad2a4cbb57-istio-token\") pod \"e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb\" (UID: \"c7d032f3-6e35-41e0-9367-9fad2a4cbb57\") " pod="openshift-ingress/e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb" Jul 13 22:37:51.151673 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:51.151606 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md6xk\" (UniqueName: \"kubernetes.io/projected/c7d032f3-6e35-41e0-9367-9fad2a4cbb57-kube-api-access-md6xk\") pod \"e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb\" (UID: \"c7d032f3-6e35-41e0-9367-9fad2a4cbb57\") " pod="openshift-ingress/e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb" Jul 13 22:37:51.151673 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:51.151629 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7d032f3-6e35-41e0-9367-9fad2a4cbb57-istio-data\") pod \"e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb\" (UID: \"c7d032f3-6e35-41e0-9367-9fad2a4cbb57\") " pod="openshift-ingress/e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb" Jul 13 22:37:51.151905 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:51.151681 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7d032f3-6e35-41e0-9367-9fad2a4cbb57-istio-envoy\") pod \"e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb\" (UID: \"c7d032f3-6e35-41e0-9367-9fad2a4cbb57\") " pod="openshift-ingress/e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb" Jul 13 22:37:51.151905 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:51.151735 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7d032f3-6e35-41e0-9367-9fad2a4cbb57-workload-socket\") pod \"e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb\" (UID: \"c7d032f3-6e35-41e0-9367-9fad2a4cbb57\") " pod="openshift-ingress/e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb" Jul 13 22:37:51.252638 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:51.252605 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7d032f3-6e35-41e0-9367-9fad2a4cbb57-workload-certs\") pod \"e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb\" (UID: \"c7d032f3-6e35-41e0-9367-9fad2a4cbb57\") " pod="openshift-ingress/e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb" Jul 13 22:37:51.252855 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:51.252647 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7d032f3-6e35-41e0-9367-9fad2a4cbb57-istio-podinfo\") pod \"e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb\" (UID: \"c7d032f3-6e35-41e0-9367-9fad2a4cbb57\") " pod="openshift-ingress/e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb" Jul 13 22:37:51.252855 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:51.252667 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7d032f3-6e35-41e0-9367-9fad2a4cbb57-credential-socket\") pod \"e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb\" (UID: \"c7d032f3-6e35-41e0-9367-9fad2a4cbb57\") " pod="openshift-ingress/e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb" Jul 13 22:37:51.252855 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:51.252721 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7d032f3-6e35-41e0-9367-9fad2a4cbb57-istio-token\") pod \"e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb\" (UID: \"c7d032f3-6e35-41e0-9367-9fad2a4cbb57\") " pod="openshift-ingress/e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb" Jul 13 22:37:51.252855 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:51.252754 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md6xk\" (UniqueName: \"kubernetes.io/projected/c7d032f3-6e35-41e0-9367-9fad2a4cbb57-kube-api-access-md6xk\") pod \"e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb\" (UID: \"c7d032f3-6e35-41e0-9367-9fad2a4cbb57\") " pod="openshift-ingress/e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb" Jul 13 22:37:51.252855 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:51.252792 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7d032f3-6e35-41e0-9367-9fad2a4cbb57-istio-data\") pod \"e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb\" (UID: \"c7d032f3-6e35-41e0-9367-9fad2a4cbb57\") " pod="openshift-ingress/e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb" Jul 13 22:37:51.252855 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:51.252820 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7d032f3-6e35-41e0-9367-9fad2a4cbb57-istio-envoy\") pod \"e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb\" (UID: \"c7d032f3-6e35-41e0-9367-9fad2a4cbb57\") " pod="openshift-ingress/e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb" Jul 13 22:37:51.253169 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:51.252859 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7d032f3-6e35-41e0-9367-9fad2a4cbb57-workload-socket\") pod \"e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb\" (UID: \"c7d032f3-6e35-41e0-9367-9fad2a4cbb57\") " pod="openshift-ingress/e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb" Jul 13 22:37:51.253169 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:51.252926 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7d032f3-6e35-41e0-9367-9fad2a4cbb57-istiod-ca-cert\") pod \"e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb\" (UID: \"c7d032f3-6e35-41e0-9367-9fad2a4cbb57\") " pod="openshift-ingress/e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb" Jul 13 22:37:51.253169 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:51.253073 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7d032f3-6e35-41e0-9367-9fad2a4cbb57-workload-certs\") pod \"e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb\" (UID: \"c7d032f3-6e35-41e0-9367-9fad2a4cbb57\") " pod="openshift-ingress/e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb" Jul 13 22:37:51.253364 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:51.253330 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7d032f3-6e35-41e0-9367-9fad2a4cbb57-istio-data\") pod \"e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb\" (UID: \"c7d032f3-6e35-41e0-9367-9fad2a4cbb57\") " pod="openshift-ingress/e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb" Jul 13 22:37:51.253364 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:51.253347 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7d032f3-6e35-41e0-9367-9fad2a4cbb57-credential-socket\") pod \"e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb\" (UID: \"c7d032f3-6e35-41e0-9367-9fad2a4cbb57\") " pod="openshift-ingress/e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb" Jul 13 22:37:51.253478 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:51.253440 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7d032f3-6e35-41e0-9367-9fad2a4cbb57-workload-socket\") pod \"e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb\" (UID: \"c7d032f3-6e35-41e0-9367-9fad2a4cbb57\") " pod="openshift-ingress/e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb" Jul 13 22:37:51.253983 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:51.253952 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7d032f3-6e35-41e0-9367-9fad2a4cbb57-istiod-ca-cert\") pod \"e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb\" (UID: \"c7d032f3-6e35-41e0-9367-9fad2a4cbb57\") " pod="openshift-ingress/e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb" Jul 13 22:37:51.255653 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:51.255634 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7d032f3-6e35-41e0-9367-9fad2a4cbb57-istio-envoy\") pod \"e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb\" (UID: \"c7d032f3-6e35-41e0-9367-9fad2a4cbb57\") " pod="openshift-ingress/e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb" Jul 13 22:37:51.256061 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:51.256045 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7d032f3-6e35-41e0-9367-9fad2a4cbb57-istio-podinfo\") pod \"e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb\" (UID: \"c7d032f3-6e35-41e0-9367-9fad2a4cbb57\") " pod="openshift-ingress/e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb" Jul 13 22:37:51.262555 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:51.262524 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7d032f3-6e35-41e0-9367-9fad2a4cbb57-istio-token\") pod \"e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb\" (UID: \"c7d032f3-6e35-41e0-9367-9fad2a4cbb57\") " pod="openshift-ingress/e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb" Jul 13 22:37:51.262555 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:51.262531 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-md6xk\" (UniqueName: \"kubernetes.io/projected/c7d032f3-6e35-41e0-9367-9fad2a4cbb57-kube-api-access-md6xk\") pod \"e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb\" (UID: \"c7d032f3-6e35-41e0-9367-9fad2a4cbb57\") " pod="openshift-ingress/e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb" Jul 13 22:37:51.396086 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:51.396055 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb" Jul 13 22:37:51.551719 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:51.551686 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb"] Jul 13 22:37:51.552674 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:37:51.552644 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7d032f3_6e35_41e0_9367_9fad2a4cbb57.slice/crio-456af86bddc59a86664f0c9775f1342bb88c32b7a0c2776d8c13f3ea252609ef WatchSource:0}: Error finding container 456af86bddc59a86664f0c9775f1342bb88c32b7a0c2776d8c13f3ea252609ef: Status 404 returned error can't find the container with id 456af86bddc59a86664f0c9775f1342bb88c32b7a0c2776d8c13f3ea252609ef Jul 13 22:37:51.555269 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:51.555229 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:37:51.555350 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:51.555317 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:37:51.555386 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:51.555359 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:37:51.867436 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:51.867341 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb" event={"ID":"c7d032f3-6e35-41e0-9367-9fad2a4cbb57","Type":"ContainerStarted","Data":"a84bffeb93afef29ad4cf00b906a84ef5ceee95293a960241f29f085c1fc9147"} Jul 13 22:37:51.867436 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:51.867407 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb" event={"ID":"c7d032f3-6e35-41e0-9367-9fad2a4cbb57","Type":"ContainerStarted","Data":"456af86bddc59a86664f0c9775f1342bb88c32b7a0c2776d8c13f3ea252609ef"} Jul 13 22:37:51.886592 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:51.886534 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb" podStartSLOduration=0.886513848 podStartE2EDuration="886.513848ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:37:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:37:51.885928897 +0000 UTC m=+2277.879683213" watchObservedRunningTime="2026-07-13 22:37:51.886513848 +0000 UTC m=+2277.880268143" Jul 13 22:37:52.396826 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:52.396788 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb" Jul 13 22:37:52.401759 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:52.401729 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb" Jul 13 22:37:52.873092 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:52.873062 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb" Jul 13 22:37:52.873951 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:52.873932 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb" Jul 13 22:37:59.821709 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:59.821605 2555 patch_prober.go:28] interesting pod/e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Jul 13 22:37:59.821709 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:59.821690 2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5" podUID="135404e9-ef6b-4702-8fd6-2642fee452f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Jul 13 22:37:59.924841 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:59.924804 2555 generic.go:356] "Generic (PLEG): container finished" podID="135404e9-ef6b-4702-8fd6-2642fee452f2" containerID="6fd2b6fe89b3ed6de6cb84c162f3c0fa47b5c7c4afa427cf417bc30df9141e44" exitCode=0 Jul 13 22:37:59.925018 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:59.924880 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5" event={"ID":"135404e9-ef6b-4702-8fd6-2642fee452f2","Type":"ContainerDied","Data":"6fd2b6fe89b3ed6de6cb84c162f3c0fa47b5c7c4afa427cf417bc30df9141e44"} Jul 13 22:37:59.925326 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:59.925310 2555 scope.go:117] "RemoveContainer" containerID="6fd2b6fe89b3ed6de6cb84c162f3c0fa47b5c7c4afa427cf417bc30df9141e44" Jul 13 22:37:59.926312 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:59.926282 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:37:59.926416 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:59.926345 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:37:59.926416 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:37:59.926377 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:38:00.941601 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:00.941550 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5" event={"ID":"135404e9-ef6b-4702-8fd6-2642fee452f2","Type":"ContainerStarted","Data":"95cc3a733b9ab207ffefef57c25ad0932a7de8c374c83f531404e0cc9f4f7c08"} Jul 13 22:38:01.111335 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:01.111298 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5" Jul 13 22:38:01.111645 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:01.111500 2555 patch_prober.go:28] interesting pod/e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Jul 13 22:38:01.111823 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:01.111797 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5" podUID="135404e9-ef6b-4702-8fd6-2642fee452f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Jul 13 22:38:02.111650 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:02.111595 2555 patch_prober.go:28] interesting pod/e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Jul 13 22:38:02.112271 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:02.112237 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5" podUID="135404e9-ef6b-4702-8fd6-2642fee452f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Jul 13 22:38:03.111866 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:03.111813 2555 patch_prober.go:28] interesting pod/e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Jul 13 22:38:03.112391 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:03.111909 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5" podUID="135404e9-ef6b-4702-8fd6-2642fee452f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Jul 13 22:38:04.113189 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:04.113145 2555 patch_prober.go:28] interesting pod/e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Jul 13 22:38:04.113684 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:04.113233 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5" podUID="135404e9-ef6b-4702-8fd6-2642fee452f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Jul 13 22:38:05.116037 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:05.116005 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5" Jul 13 22:38:05.116517 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:05.116319 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5" Jul 13 22:38:05.117023 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:05.117002 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5" Jul 13 22:38:06.270407 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:06.270368 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fe823c-h46d6"] Jul 13 22:38:06.274888 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:06.274869 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fe823c-h46d6" Jul 13 22:38:06.277892 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:06.277865 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k5z5d\"" Jul 13 22:38:06.278476 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:06.278458 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 13 22:38:06.289771 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:06.289745 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fe823c-h46d6"] Jul 13 22:38:06.393776 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:06.393733 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba924c2c-8797-4fba-ba5d-ed5d7c8c108e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fe823c-h46d6\" (UID: \"ba924c2c-8797-4fba-ba5d-ed5d7c8c108e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fe823c-h46d6" Jul 13 22:38:06.495392 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:06.495350 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba924c2c-8797-4fba-ba5d-ed5d7c8c108e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fe823c-h46d6\" (UID: \"ba924c2c-8797-4fba-ba5d-ed5d7c8c108e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fe823c-h46d6" Jul 13 22:38:06.495983 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:06.495961 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba924c2c-8797-4fba-ba5d-ed5d7c8c108e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fe823c-h46d6\" (UID: \"ba924c2c-8797-4fba-ba5d-ed5d7c8c108e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fe823c-h46d6" Jul 13 22:38:06.585797 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:06.585709 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fe823c-h46d6" Jul 13 22:38:06.731294 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:06.731263 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fe823c-h46d6"] Jul 13 22:38:06.733586 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:38:06.733539 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba924c2c_8797_4fba_ba5d_ed5d7c8c108e.slice/crio-e4fa4115f3d7d27ffb7e9d10e71d606750361deba318d854d370e9b3856e7a5f WatchSource:0}: Error finding container e4fa4115f3d7d27ffb7e9d10e71d606750361deba318d854d370e9b3856e7a5f: Status 404 returned error can't find the container with id e4fa4115f3d7d27ffb7e9d10e71d606750361deba318d854d370e9b3856e7a5f Jul 13 22:38:06.983456 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:06.983421 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fe823c-h46d6" event={"ID":"ba924c2c-8797-4fba-ba5d-ed5d7c8c108e","Type":"ContainerStarted","Data":"e4fa4115f3d7d27ffb7e9d10e71d606750361deba318d854d370e9b3856e7a5f"} Jul 13 22:38:07.874321 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:07.874288 2555 patch_prober.go:28] interesting pod/e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Jul 13 22:38:07.874697 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:07.874345 2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb" podUID="c7d032f3-6e35-41e0-9367-9fad2a4cbb57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Jul 13 22:38:07.990379 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:07.990342 2555 generic.go:356] "Generic (PLEG): container finished" podID="ba924c2c-8797-4fba-ba5d-ed5d7c8c108e" containerID="f1318a0ea6d3e4d09bb94a154101139ec47099955e803ae93390c6fd885599b1" exitCode=0 Jul 13 22:38:07.990574 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:07.990433 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fe823c-h46d6" event={"ID":"ba924c2c-8797-4fba-ba5d-ed5d7c8c108e","Type":"ContainerDied","Data":"f1318a0ea6d3e4d09bb94a154101139ec47099955e803ae93390c6fd885599b1"} Jul 13 22:38:07.992043 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:07.992014 2555 generic.go:356] "Generic (PLEG): container finished" podID="c7d032f3-6e35-41e0-9367-9fad2a4cbb57" containerID="a84bffeb93afef29ad4cf00b906a84ef5ceee95293a960241f29f085c1fc9147" exitCode=0 Jul 13 22:38:07.992163 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:07.992096 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb" event={"ID":"c7d032f3-6e35-41e0-9367-9fad2a4cbb57","Type":"ContainerDied","Data":"a84bffeb93afef29ad4cf00b906a84ef5ceee95293a960241f29f085c1fc9147"} Jul 13 22:38:07.992624 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:07.992605 2555 scope.go:117] "RemoveContainer" containerID="a84bffeb93afef29ad4cf00b906a84ef5ceee95293a960241f29f085c1fc9147" Jul 13 22:38:07.993737 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:07.993707 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:38:07.993828 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:07.993784 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:38:07.993897 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:07.993828 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:38:09.004989 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:09.004938 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb" event={"ID":"c7d032f3-6e35-41e0-9367-9fad2a4cbb57","Type":"ContainerStarted","Data":"97ba0186ae5658cebb0e38b9772162204b3abd74299826455d2df5263c45a96a"} Jul 13 22:38:09.216477 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:09.216440 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fe823c-h46d6" Jul 13 22:38:09.325957 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:09.325858 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba924c2c-8797-4fba-ba5d-ed5d7c8c108e-maas-api-service-ca\") pod \"ba924c2c-8797-4fba-ba5d-ed5d7c8c108e\" (UID: \"ba924c2c-8797-4fba-ba5d-ed5d7c8c108e\") " Jul 13 22:38:09.326260 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:09.326191 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba924c2c-8797-4fba-ba5d-ed5d7c8c108e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ba924c2c-8797-4fba-ba5d-ed5d7c8c108e" (UID: "ba924c2c-8797-4fba-ba5d-ed5d7c8c108e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:38:09.397609 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:09.397573 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb" Jul 13 22:38:09.397947 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:09.397924 2555 patch_prober.go:28] interesting pod/e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Jul 13 22:38:09.398111 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:09.398085 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb" podUID="c7d032f3-6e35-41e0-9367-9fad2a4cbb57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Jul 13 22:38:09.427322 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:09.427283 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba924c2c-8797-4fba-ba5d-ed5d7c8c108e-maas-api-service-ca\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:38:10.016168 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:10.016137 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fe823c-h46d6" Jul 13 22:38:10.017524 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:10.017494 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fe823c-h46d6" event={"ID":"ba924c2c-8797-4fba-ba5d-ed5d7c8c108e","Type":"ContainerDied","Data":"e4fa4115f3d7d27ffb7e9d10e71d606750361deba318d854d370e9b3856e7a5f"} Jul 13 22:38:10.017705 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:10.017689 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4fa4115f3d7d27ffb7e9d10e71d606750361deba318d854d370e9b3856e7a5f" Jul 13 22:38:10.396928 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:10.396890 2555 patch_prober.go:28] interesting pod/e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Jul 13 22:38:10.397319 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:10.397286 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb" podUID="c7d032f3-6e35-41e0-9367-9fad2a4cbb57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Jul 13 22:38:11.397143 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:11.397104 2555 patch_prober.go:28] interesting pod/e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Jul 13 22:38:11.398081 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:11.397698 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb" podUID="c7d032f3-6e35-41e0-9367-9fad2a4cbb57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Jul 13 22:38:12.397109 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:12.397071 2555 patch_prober.go:28] interesting pod/e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Jul 13 22:38:12.397455 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:12.397416 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb" podUID="c7d032f3-6e35-41e0-9367-9fad2a4cbb57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Jul 13 22:38:13.401051 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:13.401018 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb" Jul 13 22:38:13.401438 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:13.401304 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb" Jul 13 22:38:13.402099 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:13.402077 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb" Jul 13 22:38:32.037512 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:32.037478 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5"] Jul 13 22:38:32.038104 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:32.037735 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5" podUID="135404e9-ef6b-4702-8fd6-2642fee452f2" containerName="istio-proxy" containerID="cri-o://95cc3a733b9ab207ffefef57c25ad0932a7de8c374c83f531404e0cc9f4f7c08" gracePeriod=30 Jul 13 22:38:32.386349 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:32.386311 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a08d3-snjpb"] Jul 13 22:38:32.386741 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:32.386726 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba924c2c-8797-4fba-ba5d-ed5d7c8c108e" containerName="revoke-keys" Jul 13 22:38:32.386741 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:32.386742 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba924c2c-8797-4fba-ba5d-ed5d7c8c108e" containerName="revoke-keys" Jul 13 22:38:32.386829 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:32.386806 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba924c2c-8797-4fba-ba5d-ed5d7c8c108e" containerName="revoke-keys" Jul 13 22:38:32.389799 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:32.389783 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a08d3-snjpb" Jul 13 22:38:32.392425 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:32.392401 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 13 22:38:32.392560 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:32.392432 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k5z5d\"" Jul 13 22:38:32.397359 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:32.397333 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a08d3-snjpb"] Jul 13 22:38:32.535257 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:32.535181 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f3d5497-c91b-4ae0-8dc6-6d1d441ab17d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2a08d3-snjpb\" (UID: \"3f3d5497-c91b-4ae0-8dc6-6d1d441ab17d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a08d3-snjpb" Jul 13 22:38:32.636060 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:32.636020 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f3d5497-c91b-4ae0-8dc6-6d1d441ab17d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2a08d3-snjpb\" (UID: \"3f3d5497-c91b-4ae0-8dc6-6d1d441ab17d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a08d3-snjpb" Jul 13 22:38:32.636731 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:32.636683 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f3d5497-c91b-4ae0-8dc6-6d1d441ab17d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2a08d3-snjpb\" (UID: \"3f3d5497-c91b-4ae0-8dc6-6d1d441ab17d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a08d3-snjpb" Jul 13 22:38:32.702760 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:32.702717 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a08d3-snjpb" Jul 13 22:38:32.838172 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:32.838146 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a08d3-snjpb"] Jul 13 22:38:32.840822 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:38:32.840792 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f3d5497_c91b_4ae0_8dc6_6d1d441ab17d.slice/crio-6defdb14ad0cb49b62ceecec89cc0039c742e073b52dafca2c040be41fcebdbc WatchSource:0}: Error finding container 6defdb14ad0cb49b62ceecec89cc0039c742e073b52dafca2c040be41fcebdbc: Status 404 returned error can't find the container with id 6defdb14ad0cb49b62ceecec89cc0039c742e073b52dafca2c040be41fcebdbc Jul 13 22:38:33.166255 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:33.166133 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a08d3-snjpb" event={"ID":"3f3d5497-c91b-4ae0-8dc6-6d1d441ab17d","Type":"ContainerStarted","Data":"60ca1b62a29cd51bf035f99873b265cb53ffecae152b3060013189d0a5fc345c"} Jul 13 22:38:33.166255 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:33.166194 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a08d3-snjpb" event={"ID":"3f3d5497-c91b-4ae0-8dc6-6d1d441ab17d","Type":"ContainerStarted","Data":"6defdb14ad0cb49b62ceecec89cc0039c742e073b52dafca2c040be41fcebdbc"} Jul 13 22:38:33.182842 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:33.182796 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a08d3-snjpb" podStartSLOduration=1.182781135 podStartE2EDuration="1.182781135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:38:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:38:33.181131964 +0000 UTC m=+2319.174886257" watchObservedRunningTime="2026-07-13 22:38:33.182781135 +0000 UTC m=+2319.176535428" Jul 13 22:38:34.173548 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:34.173507 2555 generic.go:356] "Generic (PLEG): container finished" podID="3f3d5497-c91b-4ae0-8dc6-6d1d441ab17d" containerID="60ca1b62a29cd51bf035f99873b265cb53ffecae152b3060013189d0a5fc345c" exitCode=0 Jul 13 22:38:34.173926 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:34.173588 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a08d3-snjpb" event={"ID":"3f3d5497-c91b-4ae0-8dc6-6d1d441ab17d","Type":"ContainerDied","Data":"60ca1b62a29cd51bf035f99873b265cb53ffecae152b3060013189d0a5fc345c"} Jul 13 22:38:35.119243 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:35.119179 2555 patch_prober.go:28] interesting pod/e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Jul 13 22:38:35.119427 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:35.119266 2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5" podUID="135404e9-ef6b-4702-8fd6-2642fee452f2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 13 22:38:35.328241 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:35.328195 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a08d3-snjpb" Jul 13 22:38:35.463159 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:35.463127 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f3d5497-c91b-4ae0-8dc6-6d1d441ab17d-maas-api-service-ca\") pod \"3f3d5497-c91b-4ae0-8dc6-6d1d441ab17d\" (UID: \"3f3d5497-c91b-4ae0-8dc6-6d1d441ab17d\") " Jul 13 22:38:35.463526 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:35.463503 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f3d5497-c91b-4ae0-8dc6-6d1d441ab17d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3f3d5497-c91b-4ae0-8dc6-6d1d441ab17d" (UID: "3f3d5497-c91b-4ae0-8dc6-6d1d441ab17d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:38:35.563951 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:35.563917 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f3d5497-c91b-4ae0-8dc6-6d1d441ab17d-maas-api-service-ca\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:38:36.186148 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:36.186124 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a08d3-snjpb" Jul 13 22:38:36.186334 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:36.186126 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a08d3-snjpb" event={"ID":"3f3d5497-c91b-4ae0-8dc6-6d1d441ab17d","Type":"ContainerDied","Data":"6defdb14ad0cb49b62ceecec89cc0039c742e073b52dafca2c040be41fcebdbc"} Jul 13 22:38:36.186334 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:36.186253 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6defdb14ad0cb49b62ceecec89cc0039c742e073b52dafca2c040be41fcebdbc" Jul 13 22:38:37.194815 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:37.194779 2555 generic.go:356] "Generic (PLEG): container finished" podID="135404e9-ef6b-4702-8fd6-2642fee452f2" containerID="95cc3a733b9ab207ffefef57c25ad0932a7de8c374c83f531404e0cc9f4f7c08" exitCode=0 Jul 13 22:38:37.195293 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:37.194858 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5" event={"ID":"135404e9-ef6b-4702-8fd6-2642fee452f2","Type":"ContainerDied","Data":"95cc3a733b9ab207ffefef57c25ad0932a7de8c374c83f531404e0cc9f4f7c08"} Jul 13 22:38:37.195293 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:37.194935 2555 scope.go:117] "RemoveContainer" containerID="6fd2b6fe89b3ed6de6cb84c162f3c0fa47b5c7c4afa427cf417bc30df9141e44" Jul 13 22:38:37.291736 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:37.291708 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5" Jul 13 22:38:37.381532 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:37.381504 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/135404e9-ef6b-4702-8fd6-2642fee452f2-workload-socket\") pod \"135404e9-ef6b-4702-8fd6-2642fee452f2\" (UID: \"135404e9-ef6b-4702-8fd6-2642fee452f2\") " Jul 13 22:38:37.381711 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:37.381541 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/135404e9-ef6b-4702-8fd6-2642fee452f2-istiod-ca-cert\") pod \"135404e9-ef6b-4702-8fd6-2642fee452f2\" (UID: \"135404e9-ef6b-4702-8fd6-2642fee452f2\") " Jul 13 22:38:37.381711 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:37.381578 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/135404e9-ef6b-4702-8fd6-2642fee452f2-istio-envoy\") pod \"135404e9-ef6b-4702-8fd6-2642fee452f2\" (UID: \"135404e9-ef6b-4702-8fd6-2642fee452f2\") " Jul 13 22:38:37.381711 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:37.381609 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5d9nr\" (UniqueName: \"kubernetes.io/projected/135404e9-ef6b-4702-8fd6-2642fee452f2-kube-api-access-5d9nr\") pod \"135404e9-ef6b-4702-8fd6-2642fee452f2\" (UID: \"135404e9-ef6b-4702-8fd6-2642fee452f2\") " Jul 13 22:38:37.381711 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:37.381629 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/135404e9-ef6b-4702-8fd6-2642fee452f2-workload-certs\") pod \"135404e9-ef6b-4702-8fd6-2642fee452f2\" (UID: \"135404e9-ef6b-4702-8fd6-2642fee452f2\") " Jul 13 22:38:37.381711 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:37.381645 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/135404e9-ef6b-4702-8fd6-2642fee452f2-istio-data\") pod \"135404e9-ef6b-4702-8fd6-2642fee452f2\" (UID: \"135404e9-ef6b-4702-8fd6-2642fee452f2\") " Jul 13 22:38:37.381711 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:37.381698 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/135404e9-ef6b-4702-8fd6-2642fee452f2-istio-token\") pod \"135404e9-ef6b-4702-8fd6-2642fee452f2\" (UID: \"135404e9-ef6b-4702-8fd6-2642fee452f2\") " Jul 13 22:38:37.382026 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:37.381745 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/135404e9-ef6b-4702-8fd6-2642fee452f2-credential-socket\") pod \"135404e9-ef6b-4702-8fd6-2642fee452f2\" (UID: \"135404e9-ef6b-4702-8fd6-2642fee452f2\") " Jul 13 22:38:37.382026 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:37.381785 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/135404e9-ef6b-4702-8fd6-2642fee452f2-istio-podinfo\") pod \"135404e9-ef6b-4702-8fd6-2642fee452f2\" (UID: \"135404e9-ef6b-4702-8fd6-2642fee452f2\") " Jul 13 22:38:37.382026 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:37.381845 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/135404e9-ef6b-4702-8fd6-2642fee452f2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "135404e9-ef6b-4702-8fd6-2642fee452f2" (UID: "135404e9-ef6b-4702-8fd6-2642fee452f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:38:37.382026 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:37.381969 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/135404e9-ef6b-4702-8fd6-2642fee452f2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "135404e9-ef6b-4702-8fd6-2642fee452f2" (UID: "135404e9-ef6b-4702-8fd6-2642fee452f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:38:37.382276 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:37.382039 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/135404e9-ef6b-4702-8fd6-2642fee452f2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "135404e9-ef6b-4702-8fd6-2642fee452f2" (UID: "135404e9-ef6b-4702-8fd6-2642fee452f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:38:37.382276 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:37.382149 2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/135404e9-ef6b-4702-8fd6-2642fee452f2-workload-socket\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:38:37.382276 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:37.382193 2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/135404e9-ef6b-4702-8fd6-2642fee452f2-istiod-ca-cert\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:38:37.382276 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:37.382237 2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/135404e9-ef6b-4702-8fd6-2642fee452f2-workload-certs\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:38:37.382276 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:37.382258 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/135404e9-ef6b-4702-8fd6-2642fee452f2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "135404e9-ef6b-4702-8fd6-2642fee452f2" (UID: "135404e9-ef6b-4702-8fd6-2642fee452f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:38:37.382905 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:37.382872 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/135404e9-ef6b-4702-8fd6-2642fee452f2-istio-data" (OuterVolumeSpecName: "istio-data") pod "135404e9-ef6b-4702-8fd6-2642fee452f2" (UID: "135404e9-ef6b-4702-8fd6-2642fee452f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:38:37.384040 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:37.384006 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/135404e9-ef6b-4702-8fd6-2642fee452f2-istio-token" (OuterVolumeSpecName: "istio-token") pod "135404e9-ef6b-4702-8fd6-2642fee452f2" (UID: "135404e9-ef6b-4702-8fd6-2642fee452f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:38:37.384133 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:37.384072 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/135404e9-ef6b-4702-8fd6-2642fee452f2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "135404e9-ef6b-4702-8fd6-2642fee452f2" (UID: "135404e9-ef6b-4702-8fd6-2642fee452f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 13 22:38:37.384133 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:37.384090 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/135404e9-ef6b-4702-8fd6-2642fee452f2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "135404e9-ef6b-4702-8fd6-2642fee452f2" (UID: "135404e9-ef6b-4702-8fd6-2642fee452f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:38:37.384133 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:37.384125 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/135404e9-ef6b-4702-8fd6-2642fee452f2-kube-api-access-5d9nr" (OuterVolumeSpecName: "kube-api-access-5d9nr") pod "135404e9-ef6b-4702-8fd6-2642fee452f2" (UID: "135404e9-ef6b-4702-8fd6-2642fee452f2"). InnerVolumeSpecName "kube-api-access-5d9nr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:38:37.483009 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:37.482973 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5d9nr\" (UniqueName: \"kubernetes.io/projected/135404e9-ef6b-4702-8fd6-2642fee452f2-kube-api-access-5d9nr\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:38:37.483009 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:37.483009 2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/135404e9-ef6b-4702-8fd6-2642fee452f2-istio-data\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:38:37.483009 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:37.483023 2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/135404e9-ef6b-4702-8fd6-2642fee452f2-istio-token\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:38:37.483345 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:37.483036 2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/135404e9-ef6b-4702-8fd6-2642fee452f2-credential-socket\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:38:37.483345 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:37.483049 2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/135404e9-ef6b-4702-8fd6-2642fee452f2-istio-podinfo\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:38:37.483345 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:37.483061 2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/135404e9-ef6b-4702-8fd6-2642fee452f2-istio-envoy\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:38:38.202345 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:38.202317 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5" Jul 13 22:38:38.202745 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:38.202361 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5" event={"ID":"135404e9-ef6b-4702-8fd6-2642fee452f2","Type":"ContainerDied","Data":"740b030ad9c9f35520ed58666a570e91c69ff38d48341a77c2e1a730a9d4637f"} Jul 13 22:38:38.202745 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:38.202435 2555 scope.go:117] "RemoveContainer" containerID="95cc3a733b9ab207ffefef57c25ad0932a7de8c374c83f531404e0cc9f4f7c08" Jul 13 22:38:38.226402 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:38.226373 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5"] Jul 13 22:38:38.230513 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:38.230489 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-fe823c-openshift-default-bd7d64b5c-l6qm5"] Jul 13 22:38:38.560465 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:38.560382 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="135404e9-ef6b-4702-8fd6-2642fee452f2" path="/var/lib/kubelet/pods/135404e9-ef6b-4702-8fd6-2642fee452f2/volumes" Jul 13 22:38:58.131622 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:58.131584 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb"] Jul 13 22:38:58.132066 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:58.131840 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb" podUID="c7d032f3-6e35-41e0-9367-9fad2a4cbb57" containerName="istio-proxy" containerID="cri-o://97ba0186ae5658cebb0e38b9772162204b3abd74299826455d2df5263c45a96a" gracePeriod=30 Jul 13 22:38:58.404139 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:58.404049 2555 patch_prober.go:28] interesting pod/e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 13 22:38:58.404139 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:58.404108 2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb" podUID="c7d032f3-6e35-41e0-9367-9fad2a4cbb57" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 13 22:38:58.982710 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:58.982675 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc"] Jul 13 22:38:58.983140 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:58.983121 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="135404e9-ef6b-4702-8fd6-2642fee452f2" containerName="istio-proxy" Jul 13 22:38:58.983256 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:58.983143 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="135404e9-ef6b-4702-8fd6-2642fee452f2" containerName="istio-proxy" Jul 13 22:38:58.983256 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:58.983163 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f3d5497-c91b-4ae0-8dc6-6d1d441ab17d" containerName="revoke-keys" Jul 13 22:38:58.983256 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:58.983171 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f3d5497-c91b-4ae0-8dc6-6d1d441ab17d" containerName="revoke-keys" Jul 13 22:38:58.983256 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:58.983186 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="135404e9-ef6b-4702-8fd6-2642fee452f2" containerName="istio-proxy" Jul 13 22:38:58.983256 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:58.983217 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="135404e9-ef6b-4702-8fd6-2642fee452f2" containerName="istio-proxy" Jul 13 22:38:58.983492 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:58.983320 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="135404e9-ef6b-4702-8fd6-2642fee452f2" containerName="istio-proxy" Jul 13 22:38:58.983492 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:58.983332 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="135404e9-ef6b-4702-8fd6-2642fee452f2" containerName="istio-proxy" Jul 13 22:38:58.983492 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:58.983343 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f3d5497-c91b-4ae0-8dc6-6d1d441ab17d" containerName="revoke-keys" Jul 13 22:38:58.986458 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:58.986435 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc" Jul 13 22:38:58.989953 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:58.989935 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-13bc0c-openshift-default-dockercfg-nn4qc\"" Jul 13 22:38:58.998707 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:58.998684 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc"] Jul 13 22:38:59.061320 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:59.061289 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11e7e7a8-8f97-4fb4-beba-48299459a482-istio-podinfo\") pod \"e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc\" (UID: \"11e7e7a8-8f97-4fb4-beba-48299459a482\") " pod="openshift-ingress/e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc" Jul 13 22:38:59.061320 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:59.061319 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11e7e7a8-8f97-4fb4-beba-48299459a482-istiod-ca-cert\") pod \"e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc\" (UID: \"11e7e7a8-8f97-4fb4-beba-48299459a482\") " pod="openshift-ingress/e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc" Jul 13 22:38:59.061540 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:59.061345 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6jhv\" (UniqueName: \"kubernetes.io/projected/11e7e7a8-8f97-4fb4-beba-48299459a482-kube-api-access-b6jhv\") pod \"e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc\" (UID: \"11e7e7a8-8f97-4fb4-beba-48299459a482\") " pod="openshift-ingress/e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc" Jul 13 22:38:59.061540 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:59.061394 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11e7e7a8-8f97-4fb4-beba-48299459a482-istio-token\") pod \"e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc\" (UID: \"11e7e7a8-8f97-4fb4-beba-48299459a482\") " pod="openshift-ingress/e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc" Jul 13 22:38:59.061540 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:59.061473 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e7e7a8-8f97-4fb4-beba-48299459a482-workload-socket\") pod \"e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc\" (UID: \"11e7e7a8-8f97-4fb4-beba-48299459a482\") " pod="openshift-ingress/e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc" Jul 13 22:38:59.061540 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:59.061528 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11e7e7a8-8f97-4fb4-beba-48299459a482-workload-certs\") pod \"e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc\" (UID: \"11e7e7a8-8f97-4fb4-beba-48299459a482\") " pod="openshift-ingress/e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc" Jul 13 22:38:59.061736 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:59.061554 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11e7e7a8-8f97-4fb4-beba-48299459a482-istio-envoy\") pod \"e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc\" (UID: \"11e7e7a8-8f97-4fb4-beba-48299459a482\") " pod="openshift-ingress/e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc" Jul 13 22:38:59.061736 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:59.061620 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11e7e7a8-8f97-4fb4-beba-48299459a482-istio-data\") pod \"e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc\" (UID: \"11e7e7a8-8f97-4fb4-beba-48299459a482\") " pod="openshift-ingress/e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc" Jul 13 22:38:59.061736 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:59.061648 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e7e7a8-8f97-4fb4-beba-48299459a482-credential-socket\") pod \"e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc\" (UID: \"11e7e7a8-8f97-4fb4-beba-48299459a482\") " pod="openshift-ingress/e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc" Jul 13 22:38:59.162552 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:59.162519 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e7e7a8-8f97-4fb4-beba-48299459a482-workload-socket\") pod \"e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc\" (UID: \"11e7e7a8-8f97-4fb4-beba-48299459a482\") " pod="openshift-ingress/e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc" Jul 13 22:38:59.163004 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:59.162581 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11e7e7a8-8f97-4fb4-beba-48299459a482-workload-certs\") pod \"e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc\" (UID: \"11e7e7a8-8f97-4fb4-beba-48299459a482\") " pod="openshift-ingress/e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc" Jul 13 22:38:59.163004 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:59.162699 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11e7e7a8-8f97-4fb4-beba-48299459a482-istio-envoy\") pod \"e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc\" (UID: \"11e7e7a8-8f97-4fb4-beba-48299459a482\") " pod="openshift-ingress/e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc" Jul 13 22:38:59.163004 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:59.162777 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11e7e7a8-8f97-4fb4-beba-48299459a482-istio-data\") pod \"e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc\" (UID: \"11e7e7a8-8f97-4fb4-beba-48299459a482\") " pod="openshift-ingress/e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc" Jul 13 22:38:59.163004 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:59.162801 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e7e7a8-8f97-4fb4-beba-48299459a482-credential-socket\") pod \"e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc\" (UID: \"11e7e7a8-8f97-4fb4-beba-48299459a482\") " pod="openshift-ingress/e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc" Jul 13 22:38:59.163004 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:59.162822 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11e7e7a8-8f97-4fb4-beba-48299459a482-istio-podinfo\") pod \"e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc\" (UID: \"11e7e7a8-8f97-4fb4-beba-48299459a482\") " pod="openshift-ingress/e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc" Jul 13 22:38:59.163004 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:59.162844 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11e7e7a8-8f97-4fb4-beba-48299459a482-istiod-ca-cert\") pod \"e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc\" (UID: \"11e7e7a8-8f97-4fb4-beba-48299459a482\") " pod="openshift-ingress/e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc" Jul 13 22:38:59.163004 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:59.162982 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11e7e7a8-8f97-4fb4-beba-48299459a482-workload-certs\") pod \"e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc\" (UID: \"11e7e7a8-8f97-4fb4-beba-48299459a482\") " pod="openshift-ingress/e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc" Jul 13 22:38:59.163408 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:59.163029 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6jhv\" (UniqueName: \"kubernetes.io/projected/11e7e7a8-8f97-4fb4-beba-48299459a482-kube-api-access-b6jhv\") pod \"e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc\" (UID: \"11e7e7a8-8f97-4fb4-beba-48299459a482\") " pod="openshift-ingress/e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc" Jul 13 22:38:59.163408 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:59.163080 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11e7e7a8-8f97-4fb4-beba-48299459a482-istio-token\") pod \"e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc\" (UID: \"11e7e7a8-8f97-4fb4-beba-48299459a482\") " pod="openshift-ingress/e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc" Jul 13 22:38:59.163408 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:59.163173 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e7e7a8-8f97-4fb4-beba-48299459a482-credential-socket\") pod \"e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc\" (UID: \"11e7e7a8-8f97-4fb4-beba-48299459a482\") " pod="openshift-ingress/e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc" Jul 13 22:38:59.163408 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:59.163269 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11e7e7a8-8f97-4fb4-beba-48299459a482-istio-data\") pod \"e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc\" (UID: \"11e7e7a8-8f97-4fb4-beba-48299459a482\") " pod="openshift-ingress/e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc" Jul 13 22:38:59.163596 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:59.163465 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e7e7a8-8f97-4fb4-beba-48299459a482-workload-socket\") pod \"e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc\" (UID: \"11e7e7a8-8f97-4fb4-beba-48299459a482\") " pod="openshift-ingress/e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc" Jul 13 22:38:59.163596 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:59.163560 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11e7e7a8-8f97-4fb4-beba-48299459a482-istiod-ca-cert\") pod \"e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc\" (UID: \"11e7e7a8-8f97-4fb4-beba-48299459a482\") " pod="openshift-ingress/e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc" Jul 13 22:38:59.165785 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:59.165758 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11e7e7a8-8f97-4fb4-beba-48299459a482-istio-envoy\") pod \"e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc\" (UID: \"11e7e7a8-8f97-4fb4-beba-48299459a482\") " pod="openshift-ingress/e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc" Jul 13 22:38:59.165931 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:59.165911 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11e7e7a8-8f97-4fb4-beba-48299459a482-istio-podinfo\") pod \"e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc\" (UID: \"11e7e7a8-8f97-4fb4-beba-48299459a482\") " pod="openshift-ingress/e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc" Jul 13 22:38:59.176266 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:59.176243 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11e7e7a8-8f97-4fb4-beba-48299459a482-istio-token\") pod \"e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc\" (UID: \"11e7e7a8-8f97-4fb4-beba-48299459a482\") " pod="openshift-ingress/e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc" Jul 13 22:38:59.176470 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:59.176447 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6jhv\" (UniqueName: \"kubernetes.io/projected/11e7e7a8-8f97-4fb4-beba-48299459a482-kube-api-access-b6jhv\") pod \"e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc\" (UID: \"11e7e7a8-8f97-4fb4-beba-48299459a482\") " pod="openshift-ingress/e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc" Jul 13 22:38:59.298721 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:59.298618 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc" Jul 13 22:38:59.443469 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:59.443440 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc"] Jul 13 22:38:59.444637 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:38:59.444608 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11e7e7a8_8f97_4fb4_beba_48299459a482.slice/crio-5c436818d379655c3478f365cda4953d867b38160e6c3bb4d6176210c0addc47 WatchSource:0}: Error finding container 5c436818d379655c3478f365cda4953d867b38160e6c3bb4d6176210c0addc47: Status 404 returned error can't find the container with id 5c436818d379655c3478f365cda4953d867b38160e6c3bb4d6176210c0addc47 Jul 13 22:38:59.446767 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:59.446718 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:38:59.446851 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:59.446800 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:38:59.446851 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:38:59.446842 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:39:00.334762 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:00.334721 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc" event={"ID":"11e7e7a8-8f97-4fb4-beba-48299459a482","Type":"ContainerStarted","Data":"e5d9b208968596e20310ed8d61ce005c866b49102f683877beaad2ffefcd874e"} Jul 13 22:39:00.335124 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:00.334784 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc" event={"ID":"11e7e7a8-8f97-4fb4-beba-48299459a482","Type":"ContainerStarted","Data":"5c436818d379655c3478f365cda4953d867b38160e6c3bb4d6176210c0addc47"} Jul 13 22:39:00.354659 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:00.354604 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc" podStartSLOduration=2.354586744 podStartE2EDuration="2.354586744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:38:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:39:00.353728779 +0000 UTC m=+2346.347483072" watchObservedRunningTime="2026-07-13 22:39:00.354586744 +0000 UTC m=+2346.348341083" Jul 13 22:39:01.299452 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:01.299419 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc" Jul 13 22:39:01.304606 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:01.304580 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc" Jul 13 22:39:01.341316 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:01.341280 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc" Jul 13 22:39:01.342406 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:01.342388 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc" Jul 13 22:39:03.354958 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:03.354927 2555 generic.go:356] "Generic (PLEG): container finished" podID="c7d032f3-6e35-41e0-9367-9fad2a4cbb57" containerID="97ba0186ae5658cebb0e38b9772162204b3abd74299826455d2df5263c45a96a" exitCode=0 Jul 13 22:39:03.355439 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:03.355013 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb" event={"ID":"c7d032f3-6e35-41e0-9367-9fad2a4cbb57","Type":"ContainerDied","Data":"97ba0186ae5658cebb0e38b9772162204b3abd74299826455d2df5263c45a96a"} Jul 13 22:39:03.355439 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:03.355087 2555 scope.go:117] "RemoveContainer" containerID="a84bffeb93afef29ad4cf00b906a84ef5ceee95293a960241f29f085c1fc9147" Jul 13 22:39:03.405887 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:03.405806 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb" Jul 13 22:39:03.505624 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:03.505579 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7d032f3-6e35-41e0-9367-9fad2a4cbb57-workload-certs\") pod \"c7d032f3-6e35-41e0-9367-9fad2a4cbb57\" (UID: \"c7d032f3-6e35-41e0-9367-9fad2a4cbb57\") " Jul 13 22:39:03.505624 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:03.505634 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7d032f3-6e35-41e0-9367-9fad2a4cbb57-credential-socket\") pod \"c7d032f3-6e35-41e0-9367-9fad2a4cbb57\" (UID: \"c7d032f3-6e35-41e0-9367-9fad2a4cbb57\") " Jul 13 22:39:03.505903 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:03.505681 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7d032f3-6e35-41e0-9367-9fad2a4cbb57-istio-podinfo\") pod \"c7d032f3-6e35-41e0-9367-9fad2a4cbb57\" (UID: \"c7d032f3-6e35-41e0-9367-9fad2a4cbb57\") " Jul 13 22:39:03.505903 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:03.505717 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7d032f3-6e35-41e0-9367-9fad2a4cbb57-workload-socket\") pod \"c7d032f3-6e35-41e0-9367-9fad2a4cbb57\" (UID: \"c7d032f3-6e35-41e0-9367-9fad2a4cbb57\") " Jul 13 22:39:03.505903 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:03.505770 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7d032f3-6e35-41e0-9367-9fad2a4cbb57-istiod-ca-cert\") pod \"c7d032f3-6e35-41e0-9367-9fad2a4cbb57\" (UID: \"c7d032f3-6e35-41e0-9367-9fad2a4cbb57\") " Jul 13 22:39:03.505903 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:03.505802 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7d032f3-6e35-41e0-9367-9fad2a4cbb57-istio-token\") pod \"c7d032f3-6e35-41e0-9367-9fad2a4cbb57\" (UID: \"c7d032f3-6e35-41e0-9367-9fad2a4cbb57\") " Jul 13 22:39:03.505903 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:03.505824 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7d032f3-6e35-41e0-9367-9fad2a4cbb57-istio-data\") pod \"c7d032f3-6e35-41e0-9367-9fad2a4cbb57\" (UID: \"c7d032f3-6e35-41e0-9367-9fad2a4cbb57\") " Jul 13 22:39:03.505903 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:03.505850 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-md6xk\" (UniqueName: \"kubernetes.io/projected/c7d032f3-6e35-41e0-9367-9fad2a4cbb57-kube-api-access-md6xk\") pod \"c7d032f3-6e35-41e0-9367-9fad2a4cbb57\" (UID: \"c7d032f3-6e35-41e0-9367-9fad2a4cbb57\") " Jul 13 22:39:03.505903 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:03.505893 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7d032f3-6e35-41e0-9367-9fad2a4cbb57-istio-envoy\") pod \"c7d032f3-6e35-41e0-9367-9fad2a4cbb57\" (UID: \"c7d032f3-6e35-41e0-9367-9fad2a4cbb57\") " Jul 13 22:39:03.505903 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:03.505891 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7d032f3-6e35-41e0-9367-9fad2a4cbb57-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c7d032f3-6e35-41e0-9367-9fad2a4cbb57" (UID: "c7d032f3-6e35-41e0-9367-9fad2a4cbb57"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:39:03.506338 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:03.505975 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7d032f3-6e35-41e0-9367-9fad2a4cbb57-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c7d032f3-6e35-41e0-9367-9fad2a4cbb57" (UID: "c7d032f3-6e35-41e0-9367-9fad2a4cbb57"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:39:03.506338 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:03.506054 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7d032f3-6e35-41e0-9367-9fad2a4cbb57-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c7d032f3-6e35-41e0-9367-9fad2a4cbb57" (UID: "c7d032f3-6e35-41e0-9367-9fad2a4cbb57"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:39:03.506338 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:03.506283 2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7d032f3-6e35-41e0-9367-9fad2a4cbb57-workload-certs\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:39:03.506338 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:03.506303 2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7d032f3-6e35-41e0-9367-9fad2a4cbb57-credential-socket\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:39:03.506338 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:03.506317 2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7d032f3-6e35-41e0-9367-9fad2a4cbb57-workload-socket\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:39:03.507173 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:03.507142 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7d032f3-6e35-41e0-9367-9fad2a4cbb57-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c7d032f3-6e35-41e0-9367-9fad2a4cbb57" (UID: "c7d032f3-6e35-41e0-9367-9fad2a4cbb57"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:39:03.507342 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:03.507142 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7d032f3-6e35-41e0-9367-9fad2a4cbb57-istio-data" (OuterVolumeSpecName: "istio-data") pod "c7d032f3-6e35-41e0-9367-9fad2a4cbb57" (UID: "c7d032f3-6e35-41e0-9367-9fad2a4cbb57"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:39:03.508171 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:03.508139 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c7d032f3-6e35-41e0-9367-9fad2a4cbb57-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c7d032f3-6e35-41e0-9367-9fad2a4cbb57" (UID: "c7d032f3-6e35-41e0-9367-9fad2a4cbb57"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 13 22:39:03.508282 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:03.508232 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7d032f3-6e35-41e0-9367-9fad2a4cbb57-istio-token" (OuterVolumeSpecName: "istio-token") pod "c7d032f3-6e35-41e0-9367-9fad2a4cbb57" (UID: "c7d032f3-6e35-41e0-9367-9fad2a4cbb57"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:39:03.508282 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:03.508261 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7d032f3-6e35-41e0-9367-9fad2a4cbb57-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c7d032f3-6e35-41e0-9367-9fad2a4cbb57" (UID: "c7d032f3-6e35-41e0-9367-9fad2a4cbb57"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:39:03.508373 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:03.508287 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7d032f3-6e35-41e0-9367-9fad2a4cbb57-kube-api-access-md6xk" (OuterVolumeSpecName: "kube-api-access-md6xk") pod "c7d032f3-6e35-41e0-9367-9fad2a4cbb57" (UID: "c7d032f3-6e35-41e0-9367-9fad2a4cbb57"). InnerVolumeSpecName "kube-api-access-md6xk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:39:03.607365 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:03.607334 2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7d032f3-6e35-41e0-9367-9fad2a4cbb57-istiod-ca-cert\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:39:03.607365 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:03.607362 2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7d032f3-6e35-41e0-9367-9fad2a4cbb57-istio-token\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:39:03.607567 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:03.607375 2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7d032f3-6e35-41e0-9367-9fad2a4cbb57-istio-data\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:39:03.607567 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:03.607389 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-md6xk\" (UniqueName: \"kubernetes.io/projected/c7d032f3-6e35-41e0-9367-9fad2a4cbb57-kube-api-access-md6xk\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:39:03.607567 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:03.607399 2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7d032f3-6e35-41e0-9367-9fad2a4cbb57-istio-envoy\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:39:03.607567 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:03.607410 2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7d032f3-6e35-41e0-9367-9fad2a4cbb57-istio-podinfo\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:39:04.363412 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:04.363384 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb" Jul 13 22:39:04.363913 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:04.363416 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb" event={"ID":"c7d032f3-6e35-41e0-9367-9fad2a4cbb57","Type":"ContainerDied","Data":"456af86bddc59a86664f0c9775f1342bb88c32b7a0c2776d8c13f3ea252609ef"} Jul 13 22:39:04.363913 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:04.363476 2555 scope.go:117] "RemoveContainer" containerID="97ba0186ae5658cebb0e38b9772162204b3abd74299826455d2df5263c45a96a" Jul 13 22:39:04.387143 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:04.387117 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb"] Jul 13 22:39:04.392798 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:04.392771 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-2a08d3-openshift-default-649d746665-zz6fb"] Jul 13 22:39:04.560063 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:04.560028 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7d032f3-6e35-41e0-9367-9fad2a4cbb57" path="/var/lib/kubelet/pods/c7d032f3-6e35-41e0-9367-9fad2a4cbb57/volumes" Jul 13 22:39:07.335914 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:07.335866 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp"] Jul 13 22:39:07.336295 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:07.336283 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7d032f3-6e35-41e0-9367-9fad2a4cbb57" containerName="istio-proxy" Jul 13 22:39:07.336339 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:07.336296 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7d032f3-6e35-41e0-9367-9fad2a4cbb57" containerName="istio-proxy" Jul 13 22:39:07.336374 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:07.336360 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7d032f3-6e35-41e0-9367-9fad2a4cbb57" containerName="istio-proxy" Jul 13 22:39:07.336374 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:07.336370 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7d032f3-6e35-41e0-9367-9fad2a4cbb57" containerName="istio-proxy" Jul 13 22:39:07.340982 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:07.340961 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp" Jul 13 22:39:07.343705 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:07.343685 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-bdc0cb-openshift-default-dockercfg-zgjbc\"" Jul 13 22:39:07.356117 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:07.356093 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp"] Jul 13 22:39:07.443725 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:07.443684 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6cfcba7-e72f-410a-9970-738567d220fb-credential-socket\") pod \"e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp\" (UID: \"e6cfcba7-e72f-410a-9970-738567d220fb\") " pod="openshift-ingress/e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp" Jul 13 22:39:07.443923 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:07.443743 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6cfcba7-e72f-410a-9970-738567d220fb-istio-podinfo\") pod \"e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp\" (UID: \"e6cfcba7-e72f-410a-9970-738567d220fb\") " pod="openshift-ingress/e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp" Jul 13 22:39:07.443923 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:07.443824 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6cfcba7-e72f-410a-9970-738567d220fb-istio-data\") pod \"e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp\" (UID: \"e6cfcba7-e72f-410a-9970-738567d220fb\") " pod="openshift-ingress/e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp" Jul 13 22:39:07.443923 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:07.443864 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6cfcba7-e72f-410a-9970-738567d220fb-istio-envoy\") pod \"e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp\" (UID: \"e6cfcba7-e72f-410a-9970-738567d220fb\") " pod="openshift-ingress/e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp" Jul 13 22:39:07.443923 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:07.443889 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6cfcba7-e72f-410a-9970-738567d220fb-istio-token\") pod \"e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp\" (UID: \"e6cfcba7-e72f-410a-9970-738567d220fb\") " pod="openshift-ingress/e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp" Jul 13 22:39:07.444157 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:07.443929 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6cfcba7-e72f-410a-9970-738567d220fb-workload-certs\") pod \"e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp\" (UID: \"e6cfcba7-e72f-410a-9970-738567d220fb\") " pod="openshift-ingress/e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp" Jul 13 22:39:07.444157 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:07.443956 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9svw\" (UniqueName: \"kubernetes.io/projected/e6cfcba7-e72f-410a-9970-738567d220fb-kube-api-access-w9svw\") pod \"e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp\" (UID: \"e6cfcba7-e72f-410a-9970-738567d220fb\") " pod="openshift-ingress/e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp" Jul 13 22:39:07.444157 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:07.444013 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6cfcba7-e72f-410a-9970-738567d220fb-workload-socket\") pod \"e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp\" (UID: \"e6cfcba7-e72f-410a-9970-738567d220fb\") " pod="openshift-ingress/e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp" Jul 13 22:39:07.444157 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:07.444067 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6cfcba7-e72f-410a-9970-738567d220fb-istiod-ca-cert\") pod \"e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp\" (UID: \"e6cfcba7-e72f-410a-9970-738567d220fb\") " pod="openshift-ingress/e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp" Jul 13 22:39:07.544512 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:07.544478 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6cfcba7-e72f-410a-9970-738567d220fb-istio-envoy\") pod \"e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp\" (UID: \"e6cfcba7-e72f-410a-9970-738567d220fb\") " pod="openshift-ingress/e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp" Jul 13 22:39:07.544512 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:07.544515 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6cfcba7-e72f-410a-9970-738567d220fb-istio-token\") pod \"e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp\" (UID: \"e6cfcba7-e72f-410a-9970-738567d220fb\") " pod="openshift-ingress/e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp" Jul 13 22:39:07.544759 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:07.544539 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6cfcba7-e72f-410a-9970-738567d220fb-workload-certs\") pod \"e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp\" (UID: \"e6cfcba7-e72f-410a-9970-738567d220fb\") " pod="openshift-ingress/e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp" Jul 13 22:39:07.544759 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:07.544555 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9svw\" (UniqueName: \"kubernetes.io/projected/e6cfcba7-e72f-410a-9970-738567d220fb-kube-api-access-w9svw\") pod \"e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp\" (UID: \"e6cfcba7-e72f-410a-9970-738567d220fb\") " pod="openshift-ingress/e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp" Jul 13 22:39:07.544759 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:07.544628 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6cfcba7-e72f-410a-9970-738567d220fb-workload-socket\") pod \"e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp\" (UID: \"e6cfcba7-e72f-410a-9970-738567d220fb\") " pod="openshift-ingress/e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp" Jul 13 22:39:07.544759 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:07.544697 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6cfcba7-e72f-410a-9970-738567d220fb-istiod-ca-cert\") pod \"e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp\" (UID: \"e6cfcba7-e72f-410a-9970-738567d220fb\") " pod="openshift-ingress/e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp" Jul 13 22:39:07.544759 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:07.544750 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6cfcba7-e72f-410a-9970-738567d220fb-credential-socket\") pod \"e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp\" (UID: \"e6cfcba7-e72f-410a-9970-738567d220fb\") " pod="openshift-ingress/e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp" Jul 13 22:39:07.545017 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:07.544777 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6cfcba7-e72f-410a-9970-738567d220fb-istio-podinfo\") pod \"e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp\" (UID: \"e6cfcba7-e72f-410a-9970-738567d220fb\") " pod="openshift-ingress/e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp" Jul 13 22:39:07.545017 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:07.544861 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6cfcba7-e72f-410a-9970-738567d220fb-istio-data\") pod \"e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp\" (UID: \"e6cfcba7-e72f-410a-9970-738567d220fb\") " pod="openshift-ingress/e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp" Jul 13 22:39:07.545017 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:07.544983 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6cfcba7-e72f-410a-9970-738567d220fb-workload-certs\") pod \"e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp\" (UID: \"e6cfcba7-e72f-410a-9970-738567d220fb\") " pod="openshift-ingress/e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp" Jul 13 22:39:07.545184 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:07.545163 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6cfcba7-e72f-410a-9970-738567d220fb-workload-socket\") pod \"e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp\" (UID: \"e6cfcba7-e72f-410a-9970-738567d220fb\") " pod="openshift-ingress/e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp" Jul 13 22:39:07.545278 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:07.545223 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6cfcba7-e72f-410a-9970-738567d220fb-istio-data\") pod \"e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp\" (UID: \"e6cfcba7-e72f-410a-9970-738567d220fb\") " pod="openshift-ingress/e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp" Jul 13 22:39:07.545347 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:07.545322 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6cfcba7-e72f-410a-9970-738567d220fb-credential-socket\") pod \"e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp\" (UID: \"e6cfcba7-e72f-410a-9970-738567d220fb\") " pod="openshift-ingress/e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp" Jul 13 22:39:07.545510 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:07.545492 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6cfcba7-e72f-410a-9970-738567d220fb-istiod-ca-cert\") pod \"e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp\" (UID: \"e6cfcba7-e72f-410a-9970-738567d220fb\") " pod="openshift-ingress/e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp" Jul 13 22:39:07.546816 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:07.546790 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6cfcba7-e72f-410a-9970-738567d220fb-istio-envoy\") pod \"e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp\" (UID: \"e6cfcba7-e72f-410a-9970-738567d220fb\") " pod="openshift-ingress/e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp" Jul 13 22:39:07.547377 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:07.547358 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6cfcba7-e72f-410a-9970-738567d220fb-istio-podinfo\") pod \"e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp\" (UID: \"e6cfcba7-e72f-410a-9970-738567d220fb\") " pod="openshift-ingress/e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp" Jul 13 22:39:07.554623 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:07.554596 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6cfcba7-e72f-410a-9970-738567d220fb-istio-token\") pod \"e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp\" (UID: \"e6cfcba7-e72f-410a-9970-738567d220fb\") " pod="openshift-ingress/e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp" Jul 13 22:39:07.554759 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:07.554658 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9svw\" (UniqueName: \"kubernetes.io/projected/e6cfcba7-e72f-410a-9970-738567d220fb-kube-api-access-w9svw\") pod \"e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp\" (UID: \"e6cfcba7-e72f-410a-9970-738567d220fb\") " pod="openshift-ingress/e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp" Jul 13 22:39:07.650932 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:07.650896 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp" Jul 13 22:39:07.800092 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:07.800053 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp"] Jul 13 22:39:07.801798 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:39:07.801664 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6cfcba7_e72f_410a_9970_738567d220fb.slice/crio-992eec300c2f7392f11b87b1559ade6b93e7d4a7f9888db580a47c8614f6e168 WatchSource:0}: Error finding container 992eec300c2f7392f11b87b1559ade6b93e7d4a7f9888db580a47c8614f6e168: Status 404 returned error can't find the container with id 992eec300c2f7392f11b87b1559ade6b93e7d4a7f9888db580a47c8614f6e168 Jul 13 22:39:07.805413 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:07.805356 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:39:07.805530 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:07.805449 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:39:07.805530 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:07.805498 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:39:08.392631 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:08.392592 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp" event={"ID":"e6cfcba7-e72f-410a-9970-738567d220fb","Type":"ContainerStarted","Data":"fac3ea81cfe658df34b460afc07c9a176e47e5daf119b3f1f555b2e6438b8b5f"} Jul 13 22:39:08.393083 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:08.392662 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp" event={"ID":"e6cfcba7-e72f-410a-9970-738567d220fb","Type":"ContainerStarted","Data":"992eec300c2f7392f11b87b1559ade6b93e7d4a7f9888db580a47c8614f6e168"} Jul 13 22:39:08.414142 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:08.414097 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp" podStartSLOduration=1.414081837 podStartE2EDuration="1.414081837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:39:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:39:08.413161841 +0000 UTC m=+2354.406916158" watchObservedRunningTime="2026-07-13 22:39:08.414081837 +0000 UTC m=+2354.407836131" Jul 13 22:39:08.651723 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:08.651615 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp" Jul 13 22:39:08.656855 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:08.656830 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp" Jul 13 22:39:09.402479 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:09.402439 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp" Jul 13 22:39:09.405127 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:09.404850 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp" Jul 13 22:39:16.343124 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:16.343082 2555 patch_prober.go:28] interesting pod/e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Jul 13 22:39:16.343579 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:16.343138 2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc" podUID="11e7e7a8-8f97-4fb4-beba-48299459a482" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Jul 13 22:39:16.458075 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:16.458041 2555 generic.go:356] "Generic (PLEG): container finished" podID="11e7e7a8-8f97-4fb4-beba-48299459a482" containerID="e5d9b208968596e20310ed8d61ce005c866b49102f683877beaad2ffefcd874e" exitCode=0 Jul 13 22:39:16.458286 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:16.458117 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc" event={"ID":"11e7e7a8-8f97-4fb4-beba-48299459a482","Type":"ContainerDied","Data":"e5d9b208968596e20310ed8d61ce005c866b49102f683877beaad2ffefcd874e"} Jul 13 22:39:16.458564 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:16.458549 2555 scope.go:117] "RemoveContainer" containerID="e5d9b208968596e20310ed8d61ce005c866b49102f683877beaad2ffefcd874e" Jul 13 22:39:16.459547 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:16.459518 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:39:16.459602 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:16.459594 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:39:16.459643 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:16.459628 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:39:17.465681 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:17.465645 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc" event={"ID":"11e7e7a8-8f97-4fb4-beba-48299459a482","Type":"ContainerStarted","Data":"a9b0d4ca58a26f4f0b8ba1f6459208212c3adedd814f4a93a839834262d1bf1c"} Jul 13 22:39:18.299642 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:18.299600 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc" Jul 13 22:39:18.299836 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:18.299738 2555 patch_prober.go:28] interesting pod/e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Jul 13 22:39:18.299836 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:18.299788 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc" podUID="11e7e7a8-8f97-4fb4-beba-48299459a482" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Jul 13 22:39:19.299885 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:19.299836 2555 patch_prober.go:28] interesting pod/e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Jul 13 22:39:19.300429 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:19.299907 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc" podUID="11e7e7a8-8f97-4fb4-beba-48299459a482" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Jul 13 22:39:20.299921 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:20.299873 2555 patch_prober.go:28] interesting pod/e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Jul 13 22:39:20.300430 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:20.299947 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc" podUID="11e7e7a8-8f97-4fb4-beba-48299459a482" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Jul 13 22:39:21.299746 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:21.299707 2555 patch_prober.go:28] interesting pod/e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Jul 13 22:39:21.299958 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:21.299775 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc" podUID="11e7e7a8-8f97-4fb4-beba-48299459a482" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Jul 13 22:39:22.303574 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:22.303535 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc" Jul 13 22:39:22.304228 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:22.304182 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc" Jul 13 22:39:22.305515 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:22.305493 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc" Jul 13 22:39:23.505126 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:23.505030 2555 generic.go:356] "Generic (PLEG): container finished" podID="e6cfcba7-e72f-410a-9970-738567d220fb" containerID="fac3ea81cfe658df34b460afc07c9a176e47e5daf119b3f1f555b2e6438b8b5f" exitCode=0 Jul 13 22:39:23.505126 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:23.505101 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp" event={"ID":"e6cfcba7-e72f-410a-9970-738567d220fb","Type":"ContainerDied","Data":"fac3ea81cfe658df34b460afc07c9a176e47e5daf119b3f1f555b2e6438b8b5f"} Jul 13 22:39:23.505834 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:23.505641 2555 scope.go:117] "RemoveContainer" containerID="fac3ea81cfe658df34b460afc07c9a176e47e5daf119b3f1f555b2e6438b8b5f" Jul 13 22:39:23.506605 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:23.506577 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:39:23.506692 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:23.506657 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:39:23.506692 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:23.506689 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 13 22:39:24.402393 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:24.402299 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp" Jul 13 22:39:24.518222 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:24.518150 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp" event={"ID":"e6cfcba7-e72f-410a-9970-738567d220fb","Type":"ContainerStarted","Data":"51f63ecbbfa4450adeb7f94f86640fd32761ca0fd3399685243e9d883caddd9b"} Jul 13 22:39:24.652029 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:24.651988 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp" Jul 13 22:39:24.653272 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:24.653183 2555 patch_prober.go:28] interesting pod/e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Jul 13 22:39:24.653354 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:24.653267 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp" podUID="e6cfcba7-e72f-410a-9970-738567d220fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Jul 13 22:39:25.651370 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:25.651328 2555 patch_prober.go:28] interesting pod/e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Jul 13 22:39:25.651846 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:25.651413 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp" podUID="e6cfcba7-e72f-410a-9970-738567d220fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Jul 13 22:39:26.651868 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:26.651835 2555 patch_prober.go:28] interesting pod/e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Jul 13 22:39:26.652270 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:26.651885 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp" podUID="e6cfcba7-e72f-410a-9970-738567d220fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Jul 13 22:39:27.651876 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:27.651840 2555 patch_prober.go:28] interesting pod/e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Jul 13 22:39:27.652271 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:27.651895 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp" podUID="e6cfcba7-e72f-410a-9970-738567d220fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Jul 13 22:39:28.655166 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:28.655130 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp" Jul 13 22:39:28.655677 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:28.655556 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp" Jul 13 22:39:28.656344 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:39:28.656323 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp" Jul 13 22:44:58.362950 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:44:58.362913 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7569-140593683007280"] Jul 13 22:44:58.363438 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:44:58.363365 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7d032f3-6e35-41e0-9367-9fad2a4cbb57" containerName="istio-proxy" Jul 13 22:44:58.363438 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:44:58.363379 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7d032f3-6e35-41e0-9367-9fad2a4cbb57" containerName="istio-proxy" Jul 13 22:44:58.366529 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:44:58.366513 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7569-140593683007280" Jul 13 22:44:58.369341 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:44:58.369317 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-bkccs\"" Jul 13 22:44:58.372925 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:44:58.372902 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7569-140593683007280"] Jul 13 22:44:58.507218 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:44:58.507163 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w72dk\" (UniqueName: \"kubernetes.io/projected/5bf4ddc6-d583-48e2-aafd-f98228679281-kube-api-access-w72dk\") pod \"test-curl-7569-140593683007280\" (UID: \"5bf4ddc6-d583-48e2-aafd-f98228679281\") " pod="openshift-ingress/test-curl-7569-140593683007280" Jul 13 22:44:58.608425 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:44:58.608393 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w72dk\" (UniqueName: \"kubernetes.io/projected/5bf4ddc6-d583-48e2-aafd-f98228679281-kube-api-access-w72dk\") pod \"test-curl-7569-140593683007280\" (UID: \"5bf4ddc6-d583-48e2-aafd-f98228679281\") " pod="openshift-ingress/test-curl-7569-140593683007280" Jul 13 22:44:58.616496 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:44:58.616432 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w72dk\" (UniqueName: \"kubernetes.io/projected/5bf4ddc6-d583-48e2-aafd-f98228679281-kube-api-access-w72dk\") pod \"test-curl-7569-140593683007280\" (UID: \"5bf4ddc6-d583-48e2-aafd-f98228679281\") " pod="openshift-ingress/test-curl-7569-140593683007280" Jul 13 22:44:58.678809 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:44:58.678775 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7569-140593683007280" Jul 13 22:44:58.827395 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:44:58.823950 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5bf4ddc6_d583_48e2_aafd_f98228679281.slice/crio-59ad69bf7ddae40d40ad7e34e00f6c37ff77ea0984c126d25e5bcdde607be7b6 WatchSource:0}: Error finding container 59ad69bf7ddae40d40ad7e34e00f6c37ff77ea0984c126d25e5bcdde607be7b6: Status 404 returned error can't find the container with id 59ad69bf7ddae40d40ad7e34e00f6c37ff77ea0984c126d25e5bcdde607be7b6 Jul 13 22:44:58.828220 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:44:58.828175 2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 13 22:44:58.829680 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:44:58.829653 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7569-140593683007280"] Jul 13 22:44:59.521264 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:44:59.521232 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7569-140593683007280" event={"ID":"5bf4ddc6-d583-48e2-aafd-f98228679281","Type":"ContainerStarted","Data":"59ad69bf7ddae40d40ad7e34e00f6c37ff77ea0984c126d25e5bcdde607be7b6"} Jul 13 22:45:00.138402 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:00.138367 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-bdc0cb-29733045-glc92"] Jul 13 22:45:00.141970 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:00.141948 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-bdc0cb-29733045-glc92" Jul 13 22:45:00.143962 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:00.143940 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29733045-tnxfk"] Jul 13 22:45:00.144446 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:00.144428 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k5z5d\"" Jul 13 22:45:00.147576 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:00.147531 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733045-tnxfk" Jul 13 22:45:00.151927 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:00.151902 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-bdc0cb-29733045-glc92"] Jul 13 22:45:00.153961 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:00.153941 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-bdc0cb-29733045-glc92" Jul 13 22:45:00.157479 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:00.156138 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29733045-tnxfk"] Jul 13 22:45:00.165004 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:00.164983 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733045-tnxfk" Jul 13 22:45:00.315071 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:00.315027 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-bdc0cb-29733045-glc92"] Jul 13 22:45:00.317360 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:45:00.317327 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfd90533_31e7_4d49_9bc9_abbfc2888f97.slice/crio-70a914737a8d81152b77c080b010b67b6aedc18f7677f45a1cb4d57ab5e6e50b WatchSource:0}: Error finding container 70a914737a8d81152b77c080b010b67b6aedc18f7677f45a1cb4d57ab5e6e50b: Status 404 returned error can't find the container with id 70a914737a8d81152b77c080b010b67b6aedc18f7677f45a1cb4d57ab5e6e50b Jul 13 22:45:00.336545 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:00.336466 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29733045-tnxfk"] Jul 13 22:45:00.339923 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:45:00.339886 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb851f61e_8264_489a_89b5_95716a64a621.slice/crio-e3519b1fe8bc0879d2ac20f3b4181ca479fc8fe34dbc3ae44e5cef68b2dfcfcc WatchSource:0}: Error finding container e3519b1fe8bc0879d2ac20f3b4181ca479fc8fe34dbc3ae44e5cef68b2dfcfcc: Status 404 returned error can't find the container with id e3519b1fe8bc0879d2ac20f3b4181ca479fc8fe34dbc3ae44e5cef68b2dfcfcc Jul 13 22:45:00.529002 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:00.528960 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733045-tnxfk" event={"ID":"b851f61e-8264-489a-89b5-95716a64a621","Type":"ContainerStarted","Data":"e3519b1fe8bc0879d2ac20f3b4181ca479fc8fe34dbc3ae44e5cef68b2dfcfcc"} Jul 13 22:45:00.530464 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:00.530439 2555 generic.go:356] "Generic (PLEG): container finished" podID="5bf4ddc6-d583-48e2-aafd-f98228679281" containerID="df44ee175b79f9ee348c6f23f17fe56e9af1cb5e9e4da51311228101219ecbca" exitCode=0 Jul 13 22:45:00.530598 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:00.530523 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7569-140593683007280" event={"ID":"5bf4ddc6-d583-48e2-aafd-f98228679281","Type":"ContainerDied","Data":"df44ee175b79f9ee348c6f23f17fe56e9af1cb5e9e4da51311228101219ecbca"} Jul 13 22:45:00.531660 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:00.531638 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-bdc0cb-29733045-glc92" event={"ID":"bfd90533-31e7-4d49-9bc9-abbfc2888f97","Type":"ContainerStarted","Data":"70a914737a8d81152b77c080b010b67b6aedc18f7677f45a1cb4d57ab5e6e50b"} Jul 13 22:45:01.539241 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:01.539194 2555 generic.go:356] "Generic (PLEG): container finished" podID="bfd90533-31e7-4d49-9bc9-abbfc2888f97" containerID="67dc6cf348a74be66c3a02e27d1c9bf13dfce04cb7356b95635923b9b579baa5" exitCode=0 Jul 13 22:45:01.539668 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:01.539303 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-bdc0cb-29733045-glc92" event={"ID":"bfd90533-31e7-4d49-9bc9-abbfc2888f97","Type":"ContainerDied","Data":"67dc6cf348a74be66c3a02e27d1c9bf13dfce04cb7356b95635923b9b579baa5"} Jul 13 22:45:01.540898 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:01.540869 2555 generic.go:356] "Generic (PLEG): container finished" podID="b851f61e-8264-489a-89b5-95716a64a621" containerID="9d0b8aa2aa0e046eee5aad7775f7fc657fbda1129df82949c50f86c8fa0bd977" exitCode=0 Jul 13 22:45:01.540990 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:01.540943 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733045-tnxfk" event={"ID":"b851f61e-8264-489a-89b5-95716a64a621","Type":"ContainerDied","Data":"9d0b8aa2aa0e046eee5aad7775f7fc657fbda1129df82949c50f86c8fa0bd977"} Jul 13 22:45:01.693697 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:01.693669 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7569-140593683007280" Jul 13 22:45:01.724689 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:01.724663 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7569-140593683007280_5bf4ddc6-d583-48e2-aafd-f98228679281/test-curl-7569-140593683007280/0.log" Jul 13 22:45:01.745668 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:01.745632 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7569-140593683007280"] Jul 13 22:45:01.749537 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:01.749510 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7569-140593683007280"] Jul 13 22:45:01.840333 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:01.840303 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w72dk\" (UniqueName: \"kubernetes.io/projected/5bf4ddc6-d583-48e2-aafd-f98228679281-kube-api-access-w72dk\") pod \"5bf4ddc6-d583-48e2-aafd-f98228679281\" (UID: \"5bf4ddc6-d583-48e2-aafd-f98228679281\") " Jul 13 22:45:01.842496 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:01.842463 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bf4ddc6-d583-48e2-aafd-f98228679281-kube-api-access-w72dk" (OuterVolumeSpecName: "kube-api-access-w72dk") pod "5bf4ddc6-d583-48e2-aafd-f98228679281" (UID: "5bf4ddc6-d583-48e2-aafd-f98228679281"). InnerVolumeSpecName "kube-api-access-w72dk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:45:01.870395 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:01.870364 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7569-140593682293424"] Jul 13 22:45:01.870755 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:01.870742 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bf4ddc6-d583-48e2-aafd-f98228679281" containerName="test-curl-7569-140593683007280" Jul 13 22:45:01.870802 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:01.870756 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bf4ddc6-d583-48e2-aafd-f98228679281" containerName="test-curl-7569-140593683007280" Jul 13 22:45:01.870843 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:01.870834 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bf4ddc6-d583-48e2-aafd-f98228679281" containerName="test-curl-7569-140593683007280" Jul 13 22:45:01.873851 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:01.873834 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7569-140593682293424" Jul 13 22:45:01.880105 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:01.880067 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7569-140593682293424"] Jul 13 22:45:01.941398 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:01.941363 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w72dk\" (UniqueName: \"kubernetes.io/projected/5bf4ddc6-d583-48e2-aafd-f98228679281-kube-api-access-w72dk\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:45:02.041999 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:02.041966 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-598w4\" (UniqueName: \"kubernetes.io/projected/cf6058fe-b0c7-4a36-9628-f5df0e1b0b97-kube-api-access-598w4\") pod \"test-curl-7569-140593682293424\" (UID: \"cf6058fe-b0c7-4a36-9628-f5df0e1b0b97\") " pod="openshift-ingress/test-curl-7569-140593682293424" Jul 13 22:45:02.143500 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:02.143464 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-598w4\" (UniqueName: \"kubernetes.io/projected/cf6058fe-b0c7-4a36-9628-f5df0e1b0b97-kube-api-access-598w4\") pod \"test-curl-7569-140593682293424\" (UID: \"cf6058fe-b0c7-4a36-9628-f5df0e1b0b97\") " pod="openshift-ingress/test-curl-7569-140593682293424" Jul 13 22:45:02.152708 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:02.152676 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-598w4\" (UniqueName: \"kubernetes.io/projected/cf6058fe-b0c7-4a36-9628-f5df0e1b0b97-kube-api-access-598w4\") pod \"test-curl-7569-140593682293424\" (UID: \"cf6058fe-b0c7-4a36-9628-f5df0e1b0b97\") " pod="openshift-ingress/test-curl-7569-140593682293424" Jul 13 22:45:02.183573 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:02.183523 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7569-140593682293424" Jul 13 22:45:02.326173 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:02.326149 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7569-140593682293424"] Jul 13 22:45:02.327471 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:45:02.327444 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcf6058fe_b0c7_4a36_9628_f5df0e1b0b97.slice/crio-f518743230422602b5ebc3880e7e7085578b2fececc501e8298f8e9773744b6b WatchSource:0}: Error finding container f518743230422602b5ebc3880e7e7085578b2fececc501e8298f8e9773744b6b: Status 404 returned error can't find the container with id f518743230422602b5ebc3880e7e7085578b2fececc501e8298f8e9773744b6b Jul 13 22:45:02.558462 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:02.558433 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7569-140593683007280" Jul 13 22:45:02.561676 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:02.561646 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bf4ddc6-d583-48e2-aafd-f98228679281" path="/var/lib/kubelet/pods/5bf4ddc6-d583-48e2-aafd-f98228679281/volumes" Jul 13 22:45:02.562086 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:02.562063 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7569-140593682293424" event={"ID":"cf6058fe-b0c7-4a36-9628-f5df0e1b0b97","Type":"ContainerStarted","Data":"f518743230422602b5ebc3880e7e7085578b2fececc501e8298f8e9773744b6b"} Jul 13 22:45:02.562179 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:02.562105 2555 scope.go:117] "RemoveContainer" containerID="df44ee175b79f9ee348c6f23f17fe56e9af1cb5e9e4da51311228101219ecbca" Jul 13 22:45:02.725551 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:02.725524 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-bdc0cb-29733045-glc92" Jul 13 22:45:02.729279 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:02.729253 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733045-tnxfk" Jul 13 22:45:03.567882 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:03.567853 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-bdc0cb-29733045-glc92" Jul 13 22:45:03.568279 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:03.567906 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-bdc0cb-29733045-glc92" event={"ID":"bfd90533-31e7-4d49-9bc9-abbfc2888f97","Type":"ContainerDied","Data":"70a914737a8d81152b77c080b010b67b6aedc18f7677f45a1cb4d57ab5e6e50b"} Jul 13 22:45:03.568279 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:03.567959 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70a914737a8d81152b77c080b010b67b6aedc18f7677f45a1cb4d57ab5e6e50b" Jul 13 22:45:03.569452 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:03.569426 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733045-tnxfk" Jul 13 22:45:03.569575 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:03.569470 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733045-tnxfk" event={"ID":"b851f61e-8264-489a-89b5-95716a64a621","Type":"ContainerDied","Data":"e3519b1fe8bc0879d2ac20f3b4181ca479fc8fe34dbc3ae44e5cef68b2dfcfcc"} Jul 13 22:45:03.569575 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:03.569505 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3519b1fe8bc0879d2ac20f3b4181ca479fc8fe34dbc3ae44e5cef68b2dfcfcc" Jul 13 22:45:03.571061 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:03.571035 2555 generic.go:356] "Generic (PLEG): container finished" podID="cf6058fe-b0c7-4a36-9628-f5df0e1b0b97" containerID="f619d047d7922c5fa7d38a5801eea243fa6bad663d3d7489364168301da77011" exitCode=0 Jul 13 22:45:03.571166 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:03.571068 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7569-140593682293424" event={"ID":"cf6058fe-b0c7-4a36-9628-f5df0e1b0b97","Type":"ContainerDied","Data":"f619d047d7922c5fa7d38a5801eea243fa6bad663d3d7489364168301da77011"} Jul 13 22:45:04.733554 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:04.733511 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7569-140593682293424" Jul 13 22:45:04.753285 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:04.753151 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7569-140593682293424_cf6058fe-b0c7-4a36-9628-f5df0e1b0b97/test-curl-7569-140593682293424/0.log" Jul 13 22:45:04.777811 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:04.777775 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7569-140593682293424"] Jul 13 22:45:04.782492 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:04.782458 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7569-140593682293424"] Jul 13 22:45:04.870167 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:04.870077 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-598w4\" (UniqueName: \"kubernetes.io/projected/cf6058fe-b0c7-4a36-9628-f5df0e1b0b97-kube-api-access-598w4\") pod \"cf6058fe-b0c7-4a36-9628-f5df0e1b0b97\" (UID: \"cf6058fe-b0c7-4a36-9628-f5df0e1b0b97\") " Jul 13 22:45:04.872325 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:04.872300 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf6058fe-b0c7-4a36-9628-f5df0e1b0b97-kube-api-access-598w4" (OuterVolumeSpecName: "kube-api-access-598w4") pod "cf6058fe-b0c7-4a36-9628-f5df0e1b0b97" (UID: "cf6058fe-b0c7-4a36-9628-f5df0e1b0b97"). InnerVolumeSpecName "kube-api-access-598w4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:45:04.930622 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:04.930579 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7569-140593682272304"] Jul 13 22:45:04.931065 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:04.931046 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfd90533-31e7-4d49-9bc9-abbfc2888f97" containerName="cleanup" Jul 13 22:45:04.931149 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:04.931069 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfd90533-31e7-4d49-9bc9-abbfc2888f97" containerName="cleanup" Jul 13 22:45:04.931149 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:04.931113 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b851f61e-8264-489a-89b5-95716a64a621" containerName="cleanup" Jul 13 22:45:04.931149 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:04.931124 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b851f61e-8264-489a-89b5-95716a64a621" containerName="cleanup" Jul 13 22:45:04.931149 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:04.931137 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf6058fe-b0c7-4a36-9628-f5df0e1b0b97" containerName="test-curl-7569-140593682293424" Jul 13 22:45:04.931149 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:04.931147 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf6058fe-b0c7-4a36-9628-f5df0e1b0b97" containerName="test-curl-7569-140593682293424" Jul 13 22:45:04.931456 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:04.931264 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfd90533-31e7-4d49-9bc9-abbfc2888f97" containerName="cleanup" Jul 13 22:45:04.931456 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:04.931280 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf6058fe-b0c7-4a36-9628-f5df0e1b0b97" containerName="test-curl-7569-140593682293424" Jul 13 22:45:04.931456 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:04.931289 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b851f61e-8264-489a-89b5-95716a64a621" containerName="cleanup" Jul 13 22:45:04.934425 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:04.934402 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7569-140593682272304" Jul 13 22:45:04.940688 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:04.940655 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7569-140593682272304"] Jul 13 22:45:04.971674 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:04.971638 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-598w4\" (UniqueName: \"kubernetes.io/projected/cf6058fe-b0c7-4a36-9628-f5df0e1b0b97-kube-api-access-598w4\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:45:05.072627 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:05.072588 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wmjw\" (UniqueName: \"kubernetes.io/projected/59cdd6e8-616e-412c-bdd3-07e370041649-kube-api-access-6wmjw\") pod \"test-curl-7569-140593682272304\" (UID: \"59cdd6e8-616e-412c-bdd3-07e370041649\") " pod="openshift-ingress/test-curl-7569-140593682272304" Jul 13 22:45:05.173545 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:05.173506 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wmjw\" (UniqueName: \"kubernetes.io/projected/59cdd6e8-616e-412c-bdd3-07e370041649-kube-api-access-6wmjw\") pod \"test-curl-7569-140593682272304\" (UID: \"59cdd6e8-616e-412c-bdd3-07e370041649\") " pod="openshift-ingress/test-curl-7569-140593682272304" Jul 13 22:45:05.183916 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:05.183884 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wmjw\" (UniqueName: \"kubernetes.io/projected/59cdd6e8-616e-412c-bdd3-07e370041649-kube-api-access-6wmjw\") pod \"test-curl-7569-140593682272304\" (UID: \"59cdd6e8-616e-412c-bdd3-07e370041649\") " pod="openshift-ingress/test-curl-7569-140593682272304" Jul 13 22:45:05.245116 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:05.245078 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7569-140593682272304" Jul 13 22:45:05.394803 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:45:05.394774 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod59cdd6e8_616e_412c_bdd3_07e370041649.slice/crio-a4445cda38ab133f6e1f5fd006fcb0f60d06e58f09e7301a8a6525533f475a3e WatchSource:0}: Error finding container a4445cda38ab133f6e1f5fd006fcb0f60d06e58f09e7301a8a6525533f475a3e: Status 404 returned error can't find the container with id a4445cda38ab133f6e1f5fd006fcb0f60d06e58f09e7301a8a6525533f475a3e Jul 13 22:45:05.394913 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:05.394831 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7569-140593682272304"] Jul 13 22:45:05.586500 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:05.586466 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f518743230422602b5ebc3880e7e7085578b2fececc501e8298f8e9773744b6b" Jul 13 22:45:05.586722 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:05.586581 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7569-140593682293424" Jul 13 22:45:05.588667 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:05.588631 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7569-140593682272304" event={"ID":"59cdd6e8-616e-412c-bdd3-07e370041649","Type":"ContainerStarted","Data":"a4445cda38ab133f6e1f5fd006fcb0f60d06e58f09e7301a8a6525533f475a3e"} Jul 13 22:45:05.599233 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:45:05.599160 2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7569-140593682293424\" is forbidden: User \"system:node:ip-10-0-133-234.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-133-234.ec2.internal' and this object" podUID="cf6058fe-b0c7-4a36-9628-f5df0e1b0b97" pod="openshift-ingress/test-curl-7569-140593682293424" Jul 13 22:45:06.560016 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:06.559982 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf6058fe-b0c7-4a36-9628-f5df0e1b0b97" path="/var/lib/kubelet/pods/cf6058fe-b0c7-4a36-9628-f5df0e1b0b97/volumes" Jul 13 22:45:06.596665 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:06.596629 2555 generic.go:356] "Generic (PLEG): container finished" podID="59cdd6e8-616e-412c-bdd3-07e370041649" containerID="713ed0238d5609e30e7695461a9f39b9c99290722ca8aa26241cf1f1f917fd23" exitCode=0 Jul 13 22:45:06.596840 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:06.596705 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7569-140593682272304" event={"ID":"59cdd6e8-616e-412c-bdd3-07e370041649","Type":"ContainerDied","Data":"713ed0238d5609e30e7695461a9f39b9c99290722ca8aa26241cf1f1f917fd23"} Jul 13 22:45:07.771993 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:07.771962 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7569-140593682272304" Jul 13 22:45:07.793338 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:07.793306 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7569-140593682272304_59cdd6e8-616e-412c-bdd3-07e370041649/test-curl-7569-140593682272304/0.log" Jul 13 22:45:07.817107 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:07.817071 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7569-140593682272304"] Jul 13 22:45:07.820469 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:07.820445 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7569-140593682272304"] Jul 13 22:45:07.901320 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:07.901276 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6wmjw\" (UniqueName: \"kubernetes.io/projected/59cdd6e8-616e-412c-bdd3-07e370041649-kube-api-access-6wmjw\") pod \"59cdd6e8-616e-412c-bdd3-07e370041649\" (UID: \"59cdd6e8-616e-412c-bdd3-07e370041649\") " Jul 13 22:45:07.903524 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:07.903483 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59cdd6e8-616e-412c-bdd3-07e370041649-kube-api-access-6wmjw" (OuterVolumeSpecName: "kube-api-access-6wmjw") pod "59cdd6e8-616e-412c-bdd3-07e370041649" (UID: "59cdd6e8-616e-412c-bdd3-07e370041649"). InnerVolumeSpecName "kube-api-access-6wmjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:45:07.938292 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:07.938257 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7569-140593682272304"] Jul 13 22:45:07.938675 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:07.938663 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59cdd6e8-616e-412c-bdd3-07e370041649" containerName="test-curl-7569-140593682272304" Jul 13 22:45:07.938716 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:07.938677 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="59cdd6e8-616e-412c-bdd3-07e370041649" containerName="test-curl-7569-140593682272304" Jul 13 22:45:07.938757 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:07.938748 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="59cdd6e8-616e-412c-bdd3-07e370041649" containerName="test-curl-7569-140593682272304" Jul 13 22:45:07.941839 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:07.941818 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7569-140593682272304" Jul 13 22:45:07.946402 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:07.946373 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7569-140593682272304"] Jul 13 22:45:08.002651 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:08.002617 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wmjw\" (UniqueName: \"kubernetes.io/projected/59cdd6e8-616e-412c-bdd3-07e370041649-kube-api-access-6wmjw\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:45:08.103530 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:08.103489 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwgjf\" (UniqueName: \"kubernetes.io/projected/ec865e0a-e9e5-4909-8bcb-e792c014e89b-kube-api-access-jwgjf\") pod \"test-curl-7569-140593682272304\" (UID: \"ec865e0a-e9e5-4909-8bcb-e792c014e89b\") " pod="openshift-ingress/test-curl-7569-140593682272304" Jul 13 22:45:08.205050 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:08.204949 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwgjf\" (UniqueName: \"kubernetes.io/projected/ec865e0a-e9e5-4909-8bcb-e792c014e89b-kube-api-access-jwgjf\") pod \"test-curl-7569-140593682272304\" (UID: \"ec865e0a-e9e5-4909-8bcb-e792c014e89b\") " pod="openshift-ingress/test-curl-7569-140593682272304" Jul 13 22:45:08.214302 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:08.214269 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwgjf\" (UniqueName: \"kubernetes.io/projected/ec865e0a-e9e5-4909-8bcb-e792c014e89b-kube-api-access-jwgjf\") pod \"test-curl-7569-140593682272304\" (UID: \"ec865e0a-e9e5-4909-8bcb-e792c014e89b\") " pod="openshift-ingress/test-curl-7569-140593682272304" Jul 13 22:45:08.252475 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:08.252426 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7569-140593682272304" Jul 13 22:45:08.391497 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:08.391463 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7569-140593682272304"] Jul 13 22:45:08.393020 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:45:08.392991 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podec865e0a_e9e5_4909_8bcb_e792c014e89b.slice/crio-49a9b848cde9a17fa96928ef08fc01e1595813c15c1ea3aaf200f889e45f3fe7 WatchSource:0}: Error finding container 49a9b848cde9a17fa96928ef08fc01e1595813c15c1ea3aaf200f889e45f3fe7: Status 404 returned error can't find the container with id 49a9b848cde9a17fa96928ef08fc01e1595813c15c1ea3aaf200f889e45f3fe7 Jul 13 22:45:08.560251 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:08.560137 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59cdd6e8-616e-412c-bdd3-07e370041649" path="/var/lib/kubelet/pods/59cdd6e8-616e-412c-bdd3-07e370041649/volumes" Jul 13 22:45:08.609836 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:08.609798 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7569-140593682272304" event={"ID":"ec865e0a-e9e5-4909-8bcb-e792c014e89b","Type":"ContainerStarted","Data":"49a9b848cde9a17fa96928ef08fc01e1595813c15c1ea3aaf200f889e45f3fe7"} Jul 13 22:45:08.611077 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:08.611058 2555 scope.go:117] "RemoveContainer" containerID="713ed0238d5609e30e7695461a9f39b9c99290722ca8aa26241cf1f1f917fd23" Jul 13 22:45:08.611237 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:08.611079 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7569-140593682272304" Jul 13 22:45:09.619683 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:09.619642 2555 generic.go:356] "Generic (PLEG): container finished" podID="ec865e0a-e9e5-4909-8bcb-e792c014e89b" containerID="a8106fb52cd32c79d32c549108e13865d19d3aeddee6219168e4088af4ae42c9" exitCode=0 Jul 13 22:45:09.620087 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:09.619713 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7569-140593682272304" event={"ID":"ec865e0a-e9e5-4909-8bcb-e792c014e89b","Type":"ContainerDied","Data":"a8106fb52cd32c79d32c549108e13865d19d3aeddee6219168e4088af4ae42c9"} Jul 13 22:45:10.774506 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:10.774479 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7569-140593682272304" Jul 13 22:45:10.794275 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:10.794244 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7569-140593682272304_ec865e0a-e9e5-4909-8bcb-e792c014e89b/test-curl-7569-140593682272304/0.log" Jul 13 22:45:10.815276 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:10.815231 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7569-140593682272304"] Jul 13 22:45:10.821119 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:10.821094 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7569-140593682272304"] Jul 13 22:45:10.931859 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:10.931771 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jwgjf\" (UniqueName: \"kubernetes.io/projected/ec865e0a-e9e5-4909-8bcb-e792c014e89b-kube-api-access-jwgjf\") pod \"ec865e0a-e9e5-4909-8bcb-e792c014e89b\" (UID: \"ec865e0a-e9e5-4909-8bcb-e792c014e89b\") " Jul 13 22:45:10.934170 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:10.934142 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec865e0a-e9e5-4909-8bcb-e792c014e89b-kube-api-access-jwgjf" (OuterVolumeSpecName: "kube-api-access-jwgjf") pod "ec865e0a-e9e5-4909-8bcb-e792c014e89b" (UID: "ec865e0a-e9e5-4909-8bcb-e792c014e89b"). InnerVolumeSpecName "kube-api-access-jwgjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:45:10.943810 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:10.943778 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7569-140593682273312"] Jul 13 22:45:10.944187 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:10.944174 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec865e0a-e9e5-4909-8bcb-e792c014e89b" containerName="test-curl-7569-140593682272304" Jul 13 22:45:10.944259 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:10.944189 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec865e0a-e9e5-4909-8bcb-e792c014e89b" containerName="test-curl-7569-140593682272304" Jul 13 22:45:10.944298 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:10.944275 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec865e0a-e9e5-4909-8bcb-e792c014e89b" containerName="test-curl-7569-140593682272304" Jul 13 22:45:10.947296 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:10.947280 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7569-140593682273312" Jul 13 22:45:10.952236 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:10.952215 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7569-140593682273312"] Jul 13 22:45:11.032574 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:11.032537 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwgjf\" (UniqueName: \"kubernetes.io/projected/ec865e0a-e9e5-4909-8bcb-e792c014e89b-kube-api-access-jwgjf\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:45:11.133766 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:11.133728 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wqg2\" (UniqueName: \"kubernetes.io/projected/d107c1bf-7c58-45d0-8584-36055b931f14-kube-api-access-4wqg2\") pod \"test-curl-7569-140593682273312\" (UID: \"d107c1bf-7c58-45d0-8584-36055b931f14\") " pod="openshift-ingress/test-curl-7569-140593682273312" Jul 13 22:45:11.234624 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:11.234534 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wqg2\" (UniqueName: \"kubernetes.io/projected/d107c1bf-7c58-45d0-8584-36055b931f14-kube-api-access-4wqg2\") pod \"test-curl-7569-140593682273312\" (UID: \"d107c1bf-7c58-45d0-8584-36055b931f14\") " pod="openshift-ingress/test-curl-7569-140593682273312" Jul 13 22:45:11.243248 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:11.243213 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wqg2\" (UniqueName: \"kubernetes.io/projected/d107c1bf-7c58-45d0-8584-36055b931f14-kube-api-access-4wqg2\") pod \"test-curl-7569-140593682273312\" (UID: \"d107c1bf-7c58-45d0-8584-36055b931f14\") " pod="openshift-ingress/test-curl-7569-140593682273312" Jul 13 22:45:11.257787 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:11.257757 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7569-140593682273312" Jul 13 22:45:11.397827 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:11.397798 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7569-140593682273312"] Jul 13 22:45:11.398822 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:45:11.398797 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd107c1bf_7c58_45d0_8584_36055b931f14.slice/crio-594cf42d45388286f4b8ac7197d1204ccbd267f785291aba2e8afd1e649677a2 WatchSource:0}: Error finding container 594cf42d45388286f4b8ac7197d1204ccbd267f785291aba2e8afd1e649677a2: Status 404 returned error can't find the container with id 594cf42d45388286f4b8ac7197d1204ccbd267f785291aba2e8afd1e649677a2 Jul 13 22:45:11.641647 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:11.641617 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7569-140593682272304" Jul 13 22:45:11.641647 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:11.641632 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49a9b848cde9a17fa96928ef08fc01e1595813c15c1ea3aaf200f889e45f3fe7" Jul 13 22:45:11.642833 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:11.642810 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7569-140593682273312" event={"ID":"d107c1bf-7c58-45d0-8584-36055b931f14","Type":"ContainerStarted","Data":"594cf42d45388286f4b8ac7197d1204ccbd267f785291aba2e8afd1e649677a2"} Jul 13 22:45:11.652046 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:45:11.652018 2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7569-140593682272304\" is forbidden: User \"system:node:ip-10-0-133-234.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-133-234.ec2.internal' and this object" podUID="ec865e0a-e9e5-4909-8bcb-e792c014e89b" pod="openshift-ingress/test-curl-7569-140593682272304" Jul 13 22:45:12.559572 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:12.559538 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec865e0a-e9e5-4909-8bcb-e792c014e89b" path="/var/lib/kubelet/pods/ec865e0a-e9e5-4909-8bcb-e792c014e89b/volumes" Jul 13 22:45:12.650637 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:12.650597 2555 generic.go:356] "Generic (PLEG): container finished" podID="d107c1bf-7c58-45d0-8584-36055b931f14" containerID="80e871fe8d88af865e351712f10d439fbc9ad726f8761d920324f70609c634f3" exitCode=0 Jul 13 22:45:12.650809 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:12.650678 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7569-140593682273312" event={"ID":"d107c1bf-7c58-45d0-8584-36055b931f14","Type":"ContainerDied","Data":"80e871fe8d88af865e351712f10d439fbc9ad726f8761d920324f70609c634f3"} Jul 13 22:45:13.798276 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:13.798246 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7569-140593682273312" Jul 13 22:45:13.819276 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:13.819251 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7569-140593682273312_d107c1bf-7c58-45d0-8584-36055b931f14/test-curl-7569-140593682273312/0.log" Jul 13 22:45:13.843627 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:13.843581 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7569-140593682273312"] Jul 13 22:45:13.849534 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:13.849510 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7569-140593682273312"] Jul 13 22:45:13.853085 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:13.853066 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4wqg2\" (UniqueName: \"kubernetes.io/projected/d107c1bf-7c58-45d0-8584-36055b931f14-kube-api-access-4wqg2\") pod \"d107c1bf-7c58-45d0-8584-36055b931f14\" (UID: \"d107c1bf-7c58-45d0-8584-36055b931f14\") " Jul 13 22:45:13.855284 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:13.855262 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d107c1bf-7c58-45d0-8584-36055b931f14-kube-api-access-4wqg2" (OuterVolumeSpecName: "kube-api-access-4wqg2") pod "d107c1bf-7c58-45d0-8584-36055b931f14" (UID: "d107c1bf-7c58-45d0-8584-36055b931f14"). InnerVolumeSpecName "kube-api-access-4wqg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:45:13.954261 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:13.954138 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wqg2\" (UniqueName: \"kubernetes.io/projected/d107c1bf-7c58-45d0-8584-36055b931f14-kube-api-access-4wqg2\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:45:13.971178 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:13.971143 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7569-140593682273312"] Jul 13 22:45:13.971555 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:13.971541 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d107c1bf-7c58-45d0-8584-36055b931f14" containerName="test-curl-7569-140593682273312" Jul 13 22:45:13.971601 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:13.971556 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d107c1bf-7c58-45d0-8584-36055b931f14" containerName="test-curl-7569-140593682273312" Jul 13 22:45:13.971640 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:13.971619 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d107c1bf-7c58-45d0-8584-36055b931f14" containerName="test-curl-7569-140593682273312" Jul 13 22:45:13.974574 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:13.974559 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7569-140593682273312" Jul 13 22:45:13.980945 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:13.980918 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7569-140593682273312"] Jul 13 22:45:14.054703 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:14.054671 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dx2t\" (UniqueName: \"kubernetes.io/projected/51830c79-2c9a-46d2-8101-7faad25404b7-kube-api-access-4dx2t\") pod \"test-curl-7569-140593682273312\" (UID: \"51830c79-2c9a-46d2-8101-7faad25404b7\") " pod="openshift-ingress/test-curl-7569-140593682273312" Jul 13 22:45:14.155163 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:14.155126 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dx2t\" (UniqueName: \"kubernetes.io/projected/51830c79-2c9a-46d2-8101-7faad25404b7-kube-api-access-4dx2t\") pod \"test-curl-7569-140593682273312\" (UID: \"51830c79-2c9a-46d2-8101-7faad25404b7\") " pod="openshift-ingress/test-curl-7569-140593682273312" Jul 13 22:45:14.163387 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:14.163359 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dx2t\" (UniqueName: \"kubernetes.io/projected/51830c79-2c9a-46d2-8101-7faad25404b7-kube-api-access-4dx2t\") pod \"test-curl-7569-140593682273312\" (UID: \"51830c79-2c9a-46d2-8101-7faad25404b7\") " pod="openshift-ingress/test-curl-7569-140593682273312" Jul 13 22:45:14.284806 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:14.284722 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7569-140593682273312" Jul 13 22:45:14.431559 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:14.431527 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7569-140593682273312"] Jul 13 22:45:14.432316 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:45:14.432293 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod51830c79_2c9a_46d2_8101_7faad25404b7.slice/crio-32fcb25c9cdf7bb910e0b184128dee706874d234acbfec7f274197235ac8d0fa WatchSource:0}: Error finding container 32fcb25c9cdf7bb910e0b184128dee706874d234acbfec7f274197235ac8d0fa: Status 404 returned error can't find the container with id 32fcb25c9cdf7bb910e0b184128dee706874d234acbfec7f274197235ac8d0fa Jul 13 22:45:14.561643 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:14.561560 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d107c1bf-7c58-45d0-8584-36055b931f14" path="/var/lib/kubelet/pods/d107c1bf-7c58-45d0-8584-36055b931f14/volumes" Jul 13 22:45:14.663748 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:14.663715 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7569-140593682273312" Jul 13 22:45:14.663973 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:14.663751 2555 scope.go:117] "RemoveContainer" containerID="80e871fe8d88af865e351712f10d439fbc9ad726f8761d920324f70609c634f3" Jul 13 22:45:14.665258 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:14.665077 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7569-140593682273312" event={"ID":"51830c79-2c9a-46d2-8101-7faad25404b7","Type":"ContainerStarted","Data":"32fcb25c9cdf7bb910e0b184128dee706874d234acbfec7f274197235ac8d0fa"} Jul 13 22:45:15.673033 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:15.672999 2555 generic.go:356] "Generic (PLEG): container finished" podID="51830c79-2c9a-46d2-8101-7faad25404b7" containerID="2a65b002333ae563de070ae00b375cb8ebcaa54a7565a3a200351cef75fecb79" exitCode=0 Jul 13 22:45:15.673535 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:15.673080 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7569-140593682273312" event={"ID":"51830c79-2c9a-46d2-8101-7faad25404b7","Type":"ContainerDied","Data":"2a65b002333ae563de070ae00b375cb8ebcaa54a7565a3a200351cef75fecb79"} Jul 13 22:45:16.831555 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:16.831530 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7569-140593682273312" Jul 13 22:45:16.850552 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:16.850516 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7569-140593682273312_51830c79-2c9a-46d2-8101-7faad25404b7/test-curl-7569-140593682273312/0.log" Jul 13 22:45:16.876720 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:16.876672 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7569-140593682273312"] Jul 13 22:45:16.879809 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:16.879782 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7569-140593682273312"] Jul 13 22:45:16.882824 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:16.882805 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4dx2t\" (UniqueName: \"kubernetes.io/projected/51830c79-2c9a-46d2-8101-7faad25404b7-kube-api-access-4dx2t\") pod \"51830c79-2c9a-46d2-8101-7faad25404b7\" (UID: \"51830c79-2c9a-46d2-8101-7faad25404b7\") " Jul 13 22:45:16.884978 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:16.884955 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51830c79-2c9a-46d2-8101-7faad25404b7-kube-api-access-4dx2t" (OuterVolumeSpecName: "kube-api-access-4dx2t") pod "51830c79-2c9a-46d2-8101-7faad25404b7" (UID: "51830c79-2c9a-46d2-8101-7faad25404b7"). InnerVolumeSpecName "kube-api-access-4dx2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:45:16.984301 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:16.984188 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dx2t\" (UniqueName: \"kubernetes.io/projected/51830c79-2c9a-46d2-8101-7faad25404b7-kube-api-access-4dx2t\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:45:17.061708 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:17.061673 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-13bc0c-tbzfk"] Jul 13 22:45:17.062056 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:17.062043 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51830c79-2c9a-46d2-8101-7faad25404b7" containerName="test-curl-7569-140593682273312" Jul 13 22:45:17.062113 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:17.062058 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="51830c79-2c9a-46d2-8101-7faad25404b7" containerName="test-curl-7569-140593682273312" Jul 13 22:45:17.062146 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:17.062126 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="51830c79-2c9a-46d2-8101-7faad25404b7" containerName="test-curl-7569-140593682273312" Jul 13 22:45:17.065175 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:17.065158 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-13bc0c-tbzfk" Jul 13 22:45:17.067644 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:17.067621 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 13 22:45:17.067764 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:17.067656 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k5z5d\"" Jul 13 22:45:17.074516 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:17.074491 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-13bc0c-tbzfk"] Jul 13 22:45:17.084892 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:17.084867 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fa0d14d-e36e-4479-b40b-364475e705ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-13bc0c-tbzfk\" (UID: \"0fa0d14d-e36e-4479-b40b-364475e705ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-13bc0c-tbzfk" Jul 13 22:45:17.185579 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:17.185539 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fa0d14d-e36e-4479-b40b-364475e705ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-13bc0c-tbzfk\" (UID: \"0fa0d14d-e36e-4479-b40b-364475e705ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-13bc0c-tbzfk" Jul 13 22:45:17.186247 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:17.186187 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fa0d14d-e36e-4479-b40b-364475e705ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-13bc0c-tbzfk\" (UID: \"0fa0d14d-e36e-4479-b40b-364475e705ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-13bc0c-tbzfk" Jul 13 22:45:17.377495 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:17.377456 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-13bc0c-tbzfk" Jul 13 22:45:17.515985 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:17.515960 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-13bc0c-tbzfk"] Jul 13 22:45:17.517850 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:45:17.517818 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fa0d14d_e36e_4479_b40b_364475e705ed.slice/crio-711f73b20c2e72f5189f190c0c57dc18e356ffd5a155778f7bec97503143bdf7 WatchSource:0}: Error finding container 711f73b20c2e72f5189f190c0c57dc18e356ffd5a155778f7bec97503143bdf7: Status 404 returned error can't find the container with id 711f73b20c2e72f5189f190c0c57dc18e356ffd5a155778f7bec97503143bdf7 Jul 13 22:45:17.687279 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:17.687169 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-13bc0c-tbzfk" event={"ID":"0fa0d14d-e36e-4479-b40b-364475e705ed","Type":"ContainerStarted","Data":"711f73b20c2e72f5189f190c0c57dc18e356ffd5a155778f7bec97503143bdf7"} Jul 13 22:45:17.688421 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:17.688403 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32fcb25c9cdf7bb910e0b184128dee706874d234acbfec7f274197235ac8d0fa" Jul 13 22:45:17.688516 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:17.688432 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7569-140593682273312" Jul 13 22:45:17.706681 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:45:17.706622 2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7569-140593682273312\" is forbidden: User \"system:node:ip-10-0-133-234.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-133-234.ec2.internal' and this object" podUID="51830c79-2c9a-46d2-8101-7faad25404b7" pod="openshift-ingress/test-curl-7569-140593682273312" Jul 13 22:45:18.559947 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:18.559859 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51830c79-2c9a-46d2-8101-7faad25404b7" path="/var/lib/kubelet/pods/51830c79-2c9a-46d2-8101-7faad25404b7/volumes" Jul 13 22:45:18.696092 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:18.696051 2555 generic.go:356] "Generic (PLEG): container finished" podID="0fa0d14d-e36e-4479-b40b-364475e705ed" containerID="52f8027bc0113805ad6c96cdc9cac555456457e704f857ddac940fc7c25c9945" exitCode=0 Jul 13 22:45:18.696277 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:18.696138 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-13bc0c-tbzfk" event={"ID":"0fa0d14d-e36e-4479-b40b-364475e705ed","Type":"ContainerDied","Data":"52f8027bc0113805ad6c96cdc9cac555456457e704f857ddac940fc7c25c9945"} Jul 13 22:45:19.845597 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:19.845572 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-13bc0c-tbzfk" Jul 13 22:45:19.910859 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:19.910822 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fa0d14d-e36e-4479-b40b-364475e705ed-maas-api-service-ca\") pod \"0fa0d14d-e36e-4479-b40b-364475e705ed\" (UID: \"0fa0d14d-e36e-4479-b40b-364475e705ed\") " Jul 13 22:45:19.911307 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:19.911278 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fa0d14d-e36e-4479-b40b-364475e705ed-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0fa0d14d-e36e-4479-b40b-364475e705ed" (UID: "0fa0d14d-e36e-4479-b40b-364475e705ed"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:45:20.012061 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:20.011971 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fa0d14d-e36e-4479-b40b-364475e705ed-maas-api-service-ca\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:45:20.711015 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:20.710988 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-13bc0c-tbzfk" Jul 13 22:45:20.711166 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:20.711027 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-13bc0c-tbzfk" event={"ID":"0fa0d14d-e36e-4479-b40b-364475e705ed","Type":"ContainerDied","Data":"711f73b20c2e72f5189f190c0c57dc18e356ffd5a155778f7bec97503143bdf7"} Jul 13 22:45:20.711166 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:20.711074 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="711f73b20c2e72f5189f190c0c57dc18e356ffd5a155778f7bec97503143bdf7" Jul 13 22:45:42.793941 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:42.793903 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc"] Jul 13 22:45:42.794583 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:42.794174 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc" podUID="11e7e7a8-8f97-4fb4-beba-48299459a482" containerName="istio-proxy" containerID="cri-o://a9b0d4ca58a26f4f0b8ba1f6459208212c3adedd814f4a93a839834262d1bf1c" gracePeriod=30 Jul 13 22:45:48.044493 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:48.044468 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc" Jul 13 22:45:48.167991 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:48.167958 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e7e7a8-8f97-4fb4-beba-48299459a482-credential-socket\") pod \"11e7e7a8-8f97-4fb4-beba-48299459a482\" (UID: \"11e7e7a8-8f97-4fb4-beba-48299459a482\") " Jul 13 22:45:48.167991 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:48.167992 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11e7e7a8-8f97-4fb4-beba-48299459a482-istio-data\") pod \"11e7e7a8-8f97-4fb4-beba-48299459a482\" (UID: \"11e7e7a8-8f97-4fb4-beba-48299459a482\") " Jul 13 22:45:48.168339 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:48.168036 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e7e7a8-8f97-4fb4-beba-48299459a482-workload-socket\") pod \"11e7e7a8-8f97-4fb4-beba-48299459a482\" (UID: \"11e7e7a8-8f97-4fb4-beba-48299459a482\") " Jul 13 22:45:48.168339 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:48.168073 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11e7e7a8-8f97-4fb4-beba-48299459a482-istio-podinfo\") pod \"11e7e7a8-8f97-4fb4-beba-48299459a482\" (UID: \"11e7e7a8-8f97-4fb4-beba-48299459a482\") " Jul 13 22:45:48.168339 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:48.168113 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11e7e7a8-8f97-4fb4-beba-48299459a482-workload-certs\") pod \"11e7e7a8-8f97-4fb4-beba-48299459a482\" (UID: \"11e7e7a8-8f97-4fb4-beba-48299459a482\") " Jul 13 22:45:48.168339 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:48.168251 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11e7e7a8-8f97-4fb4-beba-48299459a482-istiod-ca-cert\") pod \"11e7e7a8-8f97-4fb4-beba-48299459a482\" (UID: \"11e7e7a8-8f97-4fb4-beba-48299459a482\") " Jul 13 22:45:48.168339 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:48.168290 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11e7e7a8-8f97-4fb4-beba-48299459a482-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "11e7e7a8-8f97-4fb4-beba-48299459a482" (UID: "11e7e7a8-8f97-4fb4-beba-48299459a482"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:45:48.168339 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:48.168308 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11e7e7a8-8f97-4fb4-beba-48299459a482-istio-envoy\") pod \"11e7e7a8-8f97-4fb4-beba-48299459a482\" (UID: \"11e7e7a8-8f97-4fb4-beba-48299459a482\") " Jul 13 22:45:48.168640 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:48.168360 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6jhv\" (UniqueName: \"kubernetes.io/projected/11e7e7a8-8f97-4fb4-beba-48299459a482-kube-api-access-b6jhv\") pod \"11e7e7a8-8f97-4fb4-beba-48299459a482\" (UID: \"11e7e7a8-8f97-4fb4-beba-48299459a482\") " Jul 13 22:45:48.168640 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:48.168413 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11e7e7a8-8f97-4fb4-beba-48299459a482-istio-token\") pod \"11e7e7a8-8f97-4fb4-beba-48299459a482\" (UID: \"11e7e7a8-8f97-4fb4-beba-48299459a482\") " Jul 13 22:45:48.168640 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:48.168422 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11e7e7a8-8f97-4fb4-beba-48299459a482-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "11e7e7a8-8f97-4fb4-beba-48299459a482" (UID: "11e7e7a8-8f97-4fb4-beba-48299459a482"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:45:48.168640 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:48.168454 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11e7e7a8-8f97-4fb4-beba-48299459a482-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "11e7e7a8-8f97-4fb4-beba-48299459a482" (UID: "11e7e7a8-8f97-4fb4-beba-48299459a482"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:45:48.168640 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:48.168608 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11e7e7a8-8f97-4fb4-beba-48299459a482-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "11e7e7a8-8f97-4fb4-beba-48299459a482" (UID: "11e7e7a8-8f97-4fb4-beba-48299459a482"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:45:48.168870 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:48.168731 2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11e7e7a8-8f97-4fb4-beba-48299459a482-istiod-ca-cert\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:45:48.168870 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:48.168746 2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e7e7a8-8f97-4fb4-beba-48299459a482-credential-socket\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:45:48.168870 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:48.168756 2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e7e7a8-8f97-4fb4-beba-48299459a482-workload-socket\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:45:48.168870 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:48.168768 2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11e7e7a8-8f97-4fb4-beba-48299459a482-workload-certs\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:45:48.168870 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:48.168778 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11e7e7a8-8f97-4fb4-beba-48299459a482-istio-data" (OuterVolumeSpecName: "istio-data") pod "11e7e7a8-8f97-4fb4-beba-48299459a482" (UID: "11e7e7a8-8f97-4fb4-beba-48299459a482"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:45:48.170489 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:48.170468 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11e7e7a8-8f97-4fb4-beba-48299459a482-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "11e7e7a8-8f97-4fb4-beba-48299459a482" (UID: "11e7e7a8-8f97-4fb4-beba-48299459a482"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:45:48.170588 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:48.170572 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11e7e7a8-8f97-4fb4-beba-48299459a482-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "11e7e7a8-8f97-4fb4-beba-48299459a482" (UID: "11e7e7a8-8f97-4fb4-beba-48299459a482"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 13 22:45:48.171041 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:48.171019 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11e7e7a8-8f97-4fb4-beba-48299459a482-istio-token" (OuterVolumeSpecName: "istio-token") pod "11e7e7a8-8f97-4fb4-beba-48299459a482" (UID: "11e7e7a8-8f97-4fb4-beba-48299459a482"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:45:48.171144 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:48.171044 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11e7e7a8-8f97-4fb4-beba-48299459a482-kube-api-access-b6jhv" (OuterVolumeSpecName: "kube-api-access-b6jhv") pod "11e7e7a8-8f97-4fb4-beba-48299459a482" (UID: "11e7e7a8-8f97-4fb4-beba-48299459a482"). InnerVolumeSpecName "kube-api-access-b6jhv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:45:48.269626 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:48.269593 2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11e7e7a8-8f97-4fb4-beba-48299459a482-istio-podinfo\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:45:48.269626 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:48.269621 2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11e7e7a8-8f97-4fb4-beba-48299459a482-istio-envoy\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:45:48.269626 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:48.269631 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6jhv\" (UniqueName: \"kubernetes.io/projected/11e7e7a8-8f97-4fb4-beba-48299459a482-kube-api-access-b6jhv\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:45:48.269861 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:48.269642 2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11e7e7a8-8f97-4fb4-beba-48299459a482-istio-token\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:45:48.269861 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:48.269655 2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11e7e7a8-8f97-4fb4-beba-48299459a482-istio-data\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:45:48.890783 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:48.890753 2555 generic.go:356] "Generic (PLEG): container finished" podID="11e7e7a8-8f97-4fb4-beba-48299459a482" containerID="a9b0d4ca58a26f4f0b8ba1f6459208212c3adedd814f4a93a839834262d1bf1c" exitCode=0 Jul 13 22:45:48.890959 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:48.890807 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc" Jul 13 22:45:48.890959 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:48.890839 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc" event={"ID":"11e7e7a8-8f97-4fb4-beba-48299459a482","Type":"ContainerDied","Data":"a9b0d4ca58a26f4f0b8ba1f6459208212c3adedd814f4a93a839834262d1bf1c"} Jul 13 22:45:48.890959 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:48.890902 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc" event={"ID":"11e7e7a8-8f97-4fb4-beba-48299459a482","Type":"ContainerDied","Data":"5c436818d379655c3478f365cda4953d867b38160e6c3bb4d6176210c0addc47"} Jul 13 22:45:48.890959 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:48.890925 2555 scope.go:117] "RemoveContainer" containerID="a9b0d4ca58a26f4f0b8ba1f6459208212c3adedd814f4a93a839834262d1bf1c" Jul 13 22:45:48.900758 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:48.900743 2555 scope.go:117] "RemoveContainer" containerID="e5d9b208968596e20310ed8d61ce005c866b49102f683877beaad2ffefcd874e" Jul 13 22:45:48.911034 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:48.910870 2555 scope.go:117] "RemoveContainer" containerID="a9b0d4ca58a26f4f0b8ba1f6459208212c3adedd814f4a93a839834262d1bf1c" Jul 13 22:45:48.911185 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:48.911163 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc"] Jul 13 22:45:48.911185 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:45:48.911170 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9b0d4ca58a26f4f0b8ba1f6459208212c3adedd814f4a93a839834262d1bf1c\": container with ID starting with a9b0d4ca58a26f4f0b8ba1f6459208212c3adedd814f4a93a839834262d1bf1c not found: ID does not exist" containerID="a9b0d4ca58a26f4f0b8ba1f6459208212c3adedd814f4a93a839834262d1bf1c" Jul 13 22:45:48.911312 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:48.911222 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9b0d4ca58a26f4f0b8ba1f6459208212c3adedd814f4a93a839834262d1bf1c"} err="failed to get container status \"a9b0d4ca58a26f4f0b8ba1f6459208212c3adedd814f4a93a839834262d1bf1c\": rpc error: code = NotFound desc = could not find container \"a9b0d4ca58a26f4f0b8ba1f6459208212c3adedd814f4a93a839834262d1bf1c\": container with ID starting with a9b0d4ca58a26f4f0b8ba1f6459208212c3adedd814f4a93a839834262d1bf1c not found: ID does not exist" Jul 13 22:45:48.911312 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:48.911240 2555 scope.go:117] "RemoveContainer" containerID="e5d9b208968596e20310ed8d61ce005c866b49102f683877beaad2ffefcd874e" Jul 13 22:45:48.911495 ip-10-0-133-234 kubenswrapper[2555]: E0713 22:45:48.911480 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5d9b208968596e20310ed8d61ce005c866b49102f683877beaad2ffefcd874e\": container with ID starting with e5d9b208968596e20310ed8d61ce005c866b49102f683877beaad2ffefcd874e not found: ID does not exist" containerID="e5d9b208968596e20310ed8d61ce005c866b49102f683877beaad2ffefcd874e" Jul 13 22:45:48.911532 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:48.911500 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5d9b208968596e20310ed8d61ce005c866b49102f683877beaad2ffefcd874e"} err="failed to get container status \"e5d9b208968596e20310ed8d61ce005c866b49102f683877beaad2ffefcd874e\": rpc error: code = NotFound desc = could not find container \"e5d9b208968596e20310ed8d61ce005c866b49102f683877beaad2ffefcd874e\": container with ID starting with e5d9b208968596e20310ed8d61ce005c866b49102f683877beaad2ffefcd874e not found: ID does not exist" Jul 13 22:45:48.914056 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:48.914037 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-13bc0c-openshift-default-9679888c5-7lklc"] Jul 13 22:45:50.559464 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:50.559431 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11e7e7a8-8f97-4fb4-beba-48299459a482" path="/var/lib/kubelet/pods/11e7e7a8-8f97-4fb4-beba-48299459a482/volumes" Jul 13 22:45:53.656513 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:53.656469 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-bdc0cb-29733045-glc92"] Jul 13 22:45:53.662044 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:53.662019 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-bdc0cb-29733045-glc92"] Jul 13 22:45:54.560460 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:45:54.560428 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfd90533-31e7-4d49-9bc9-abbfc2888f97" path="/var/lib/kubelet/pods/bfd90533-31e7-4d49-9bc9-abbfc2888f97/volumes" Jul 13 22:46:08.882696 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:08.882655 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp"] Jul 13 22:46:08.883131 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:08.882857 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp" podUID="e6cfcba7-e72f-410a-9970-738567d220fb" containerName="istio-proxy" containerID="cri-o://51f63ecbbfa4450adeb7f94f86640fd32761ca0fd3399685243e9d883caddd9b" gracePeriod=30 Jul 13 22:46:09.565925 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:09.565886 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-64d6554466-4jltm_cfc6882c-943b-49f5-843d-490b9e6e075b/authorino/0.log" Jul 13 22:46:13.658074 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:13.658028 2555 patch_prober.go:28] interesting pod/e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 13 22:46:13.658507 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:13.658086 2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp" podUID="e6cfcba7-e72f-410a-9970-738567d220fb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 13 22:46:14.046577 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:14.046541 2555 generic.go:356] "Generic (PLEG): container finished" podID="e6cfcba7-e72f-410a-9970-738567d220fb" containerID="51f63ecbbfa4450adeb7f94f86640fd32761ca0fd3399685243e9d883caddd9b" exitCode=0 Jul 13 22:46:14.046765 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:14.046609 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp" event={"ID":"e6cfcba7-e72f-410a-9970-738567d220fb","Type":"ContainerDied","Data":"51f63ecbbfa4450adeb7f94f86640fd32761ca0fd3399685243e9d883caddd9b"} Jul 13 22:46:14.046765 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:14.046677 2555 scope.go:117] "RemoveContainer" containerID="fac3ea81cfe658df34b460afc07c9a176e47e5daf119b3f1f555b2e6438b8b5f" Jul 13 22:46:14.142307 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:14.142279 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp" Jul 13 22:46:14.166754 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:14.166723 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w9svw\" (UniqueName: \"kubernetes.io/projected/e6cfcba7-e72f-410a-9970-738567d220fb-kube-api-access-w9svw\") pod \"e6cfcba7-e72f-410a-9970-738567d220fb\" (UID: \"e6cfcba7-e72f-410a-9970-738567d220fb\") " Jul 13 22:46:14.166922 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:14.166766 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6cfcba7-e72f-410a-9970-738567d220fb-credential-socket\") pod \"e6cfcba7-e72f-410a-9970-738567d220fb\" (UID: \"e6cfcba7-e72f-410a-9970-738567d220fb\") " Jul 13 22:46:14.166922 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:14.166783 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6cfcba7-e72f-410a-9970-738567d220fb-workload-socket\") pod \"e6cfcba7-e72f-410a-9970-738567d220fb\" (UID: \"e6cfcba7-e72f-410a-9970-738567d220fb\") " Jul 13 22:46:14.166922 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:14.166909 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6cfcba7-e72f-410a-9970-738567d220fb-istio-podinfo\") pod \"e6cfcba7-e72f-410a-9970-738567d220fb\" (UID: \"e6cfcba7-e72f-410a-9970-738567d220fb\") " Jul 13 22:46:14.167086 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:14.166938 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6cfcba7-e72f-410a-9970-738567d220fb-istio-envoy\") pod \"e6cfcba7-e72f-410a-9970-738567d220fb\" (UID: \"e6cfcba7-e72f-410a-9970-738567d220fb\") " Jul 13 22:46:14.167086 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:14.166975 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6cfcba7-e72f-410a-9970-738567d220fb-istio-token\") pod \"e6cfcba7-e72f-410a-9970-738567d220fb\" (UID: \"e6cfcba7-e72f-410a-9970-738567d220fb\") " Jul 13 22:46:14.167086 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:14.167009 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6cfcba7-e72f-410a-9970-738567d220fb-istio-data\") pod \"e6cfcba7-e72f-410a-9970-738567d220fb\" (UID: \"e6cfcba7-e72f-410a-9970-738567d220fb\") " Jul 13 22:46:14.167086 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:14.167068 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6cfcba7-e72f-410a-9970-738567d220fb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e6cfcba7-e72f-410a-9970-738567d220fb" (UID: "e6cfcba7-e72f-410a-9970-738567d220fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:46:14.167343 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:14.167077 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6cfcba7-e72f-410a-9970-738567d220fb-istiod-ca-cert\") pod \"e6cfcba7-e72f-410a-9970-738567d220fb\" (UID: \"e6cfcba7-e72f-410a-9970-738567d220fb\") " Jul 13 22:46:14.167343 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:14.167098 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6cfcba7-e72f-410a-9970-738567d220fb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e6cfcba7-e72f-410a-9970-738567d220fb" (UID: "e6cfcba7-e72f-410a-9970-738567d220fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:46:14.167343 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:14.167138 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6cfcba7-e72f-410a-9970-738567d220fb-workload-certs\") pod \"e6cfcba7-e72f-410a-9970-738567d220fb\" (UID: \"e6cfcba7-e72f-410a-9970-738567d220fb\") " Jul 13 22:46:14.167493 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:14.167419 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6cfcba7-e72f-410a-9970-738567d220fb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e6cfcba7-e72f-410a-9970-738567d220fb" (UID: "e6cfcba7-e72f-410a-9970-738567d220fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:46:14.167493 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:14.167481 2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6cfcba7-e72f-410a-9970-738567d220fb-istiod-ca-cert\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:46:14.167593 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:14.167500 2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6cfcba7-e72f-410a-9970-738567d220fb-credential-socket\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:46:14.167593 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:14.167515 2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6cfcba7-e72f-410a-9970-738567d220fb-workload-socket\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:46:14.167700 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:14.167678 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6cfcba7-e72f-410a-9970-738567d220fb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e6cfcba7-e72f-410a-9970-738567d220fb" (UID: "e6cfcba7-e72f-410a-9970-738567d220fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:46:14.167880 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:14.167850 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6cfcba7-e72f-410a-9970-738567d220fb-istio-data" (OuterVolumeSpecName: "istio-data") pod "e6cfcba7-e72f-410a-9970-738567d220fb" (UID: "e6cfcba7-e72f-410a-9970-738567d220fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:46:14.169296 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:14.169271 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6cfcba7-e72f-410a-9970-738567d220fb-kube-api-access-w9svw" (OuterVolumeSpecName: "kube-api-access-w9svw") pod "e6cfcba7-e72f-410a-9970-738567d220fb" (UID: "e6cfcba7-e72f-410a-9970-738567d220fb"). InnerVolumeSpecName "kube-api-access-w9svw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:46:14.169778 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:14.169762 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6cfcba7-e72f-410a-9970-738567d220fb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e6cfcba7-e72f-410a-9970-738567d220fb" (UID: "e6cfcba7-e72f-410a-9970-738567d220fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:46:14.169870 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:14.169858 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6cfcba7-e72f-410a-9970-738567d220fb-istio-token" (OuterVolumeSpecName: "istio-token") pod "e6cfcba7-e72f-410a-9970-738567d220fb" (UID: "e6cfcba7-e72f-410a-9970-738567d220fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:46:14.169985 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:14.169968 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e6cfcba7-e72f-410a-9970-738567d220fb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e6cfcba7-e72f-410a-9970-738567d220fb" (UID: "e6cfcba7-e72f-410a-9970-738567d220fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 13 22:46:14.186370 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:14.186281 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-5mxzw_a5adea24-b5bc-4fc6-bedd-9357092f0205/manager/0.log" Jul 13 22:46:14.268595 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:14.268498 2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6cfcba7-e72f-410a-9970-738567d220fb-workload-certs\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:46:14.268595 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:14.268533 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9svw\" (UniqueName: \"kubernetes.io/projected/e6cfcba7-e72f-410a-9970-738567d220fb-kube-api-access-w9svw\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:46:14.268595 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:14.268542 2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6cfcba7-e72f-410a-9970-738567d220fb-istio-podinfo\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:46:14.268595 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:14.268553 2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6cfcba7-e72f-410a-9970-738567d220fb-istio-envoy\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:46:14.268595 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:14.268562 2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6cfcba7-e72f-410a-9970-738567d220fb-istio-token\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:46:14.268595 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:14.268571 2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6cfcba7-e72f-410a-9970-738567d220fb-istio-data\") on node \"ip-10-0-133-234.ec2.internal\" DevicePath \"\"" Jul 13 22:46:14.305474 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:14.305441 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-5k88d_dba67c81-79d8-4b45-8382-e7f9bfe110dc/manager/0.log" Jul 13 22:46:14.520373 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:14.520295 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-xmzbx_77499f45-4453-4263-aa49-e3a9a534e134/server/0.log" Jul 13 22:46:14.642949 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:14.642870 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-rn2vl_4035fc64-bb79-42c7-9d2c-a95f42177716/manager/0.log" Jul 13 22:46:14.750563 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:14.750530 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67ddb65fc-dwcsd_2f424140-dd4b-4331-8e78-cd10ee0f70e2/copy-manifests/0.log" Jul 13 22:46:14.756554 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:14.756527 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67ddb65fc-dwcsd_2f424140-dd4b-4331-8e78-cd10ee0f70e2/manager/0.log" Jul 13 22:46:15.053699 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:15.053668 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp" Jul 13 22:46:15.053699 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:15.053688 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp" event={"ID":"e6cfcba7-e72f-410a-9970-738567d220fb","Type":"ContainerDied","Data":"992eec300c2f7392f11b87b1559ade6b93e7d4a7f9888db580a47c8614f6e168"} Jul 13 22:46:15.053935 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:15.053746 2555 scope.go:117] "RemoveContainer" containerID="51f63ecbbfa4450adeb7f94f86640fd32761ca0fd3399685243e9d883caddd9b" Jul 13 22:46:15.072387 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:15.072358 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp"] Jul 13 22:46:15.076287 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:15.076263 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-bdc0cb-openshift-default-7c49876967-j9gfp"] Jul 13 22:46:15.482140 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:15.482110 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29733015-5wsdj_fd8a23da-b88e-4e5f-8d41-761484e073d6/cleanup/0.log" Jul 13 22:46:15.591978 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:15.591950 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29733030-mqz8r_5a20a0ca-1d06-4159-b5fd-e61a44112d6d/cleanup/0.log" Jul 13 22:46:15.703173 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:15.703142 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29733045-tnxfk_b851f61e-8264-489a-89b5-95716a64a621/cleanup/0.log" Jul 13 22:46:15.812774 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:15.812687 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-a399a528-mnt2q_771d2883-dbbb-4d75-ace3-a6b36abb9e8e/revoke-keys/0.log" Jul 13 22:46:15.922071 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:15.922038 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-d7fae36e-4lmw9_0e65cf22-17d8-4fb6-bbdf-67cd4fd4d296/revoke-keys/0.log" Jul 13 22:46:16.030711 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:16.030678 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-a-fe823c-h46d6_ba924c2c-8797-4fba-ba5d-ed5d7c8c108e/revoke-keys/0.log" Jul 13 22:46:16.139218 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:16.139169 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-2a08d3-snjpb_3f3d5497-c91b-4ae0-8dc6-6d1d441ab17d/revoke-keys/0.log" Jul 13 22:46:16.244065 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:16.244035 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-derive-0081b233-mtb4m_53051199-f7d4-4517-baf7-603ab478340f/revoke-keys/0.log" Jul 13 22:46:16.352570 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:16.352540 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-a-7da020-pn5mq_976c57b7-04ab-4c1d-9c55-5e974dbb0417/revoke-keys/0.log" Jul 13 22:46:16.459629 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:16.459546 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-b-a502db-gzrz2_af30161a-8947-484f-b8b7-2d528d7c7d09/revoke-keys/0.log" Jul 13 22:46:16.559955 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:16.559917 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6cfcba7-e72f-410a-9970-738567d220fb" path="/var/lib/kubelet/pods/e6cfcba7-e72f-410a-9970-738567d220fb/volumes" Jul 13 22:46:16.568568 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:16.568549 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-migrate-65448f9f-7m9w2_d47e57de-4b10-40f8-99c4-b127c59c8930/revoke-keys/0.log" Jul 13 22:46:16.680300 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:16.680271 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-2fa7da44-h2x7p_463cd8a3-f1c4-4389-b57e-bc854374fe4c/revoke-keys/0.log" Jul 13 22:46:16.789378 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:16.789291 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-86e22e9e-9dj4r_1010f23b-7bcf-4a62-bd05-f070208d24c2/revoke-keys/0.log" Jul 13 22:46:16.897371 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:16.897340 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-a-13bc0c-tbzfk_0fa0d14d-e36e-4479-b40b-364475e705ed/revoke-keys/0.log" Jul 13 22:46:17.868231 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:17.868165 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k5bfz_6e63fc16-f858-4f3c-84e5-3f89699f7250/util/0.log" Jul 13 22:46:17.873996 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:17.873969 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k5bfz_6e63fc16-f858-4f3c-84e5-3f89699f7250/pull/0.log" Jul 13 22:46:17.878821 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:17.878804 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k5bfz_6e63fc16-f858-4f3c-84e5-3f89699f7250/extract/0.log" Jul 13 22:46:17.985939 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:17.985913 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xxfwl_6803f4ff-7912-492d-8540-6bc0c47650dd/util/0.log" Jul 13 22:46:17.991065 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:17.991039 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xxfwl_6803f4ff-7912-492d-8540-6bc0c47650dd/pull/0.log" Jul 13 22:46:17.995970 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:17.995949 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xxfwl_6803f4ff-7912-492d-8540-6bc0c47650dd/extract/0.log" Jul 13 22:46:18.106413 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:18.106389 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736cmcr_4387c748-a3eb-44ee-9016-2688537cf27f/util/0.log" Jul 13 22:46:18.112341 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:18.112310 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736cmcr_4387c748-a3eb-44ee-9016-2688537cf27f/pull/0.log" Jul 13 22:46:18.118091 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:18.118073 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736cmcr_4387c748-a3eb-44ee-9016-2688537cf27f/extract/0.log" Jul 13 22:46:18.228174 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:18.228143 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gx9k4_464555f7-8a36-4a47-8342-dfb9dbd5bae3/util/0.log" Jul 13 22:46:18.233906 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:18.233884 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gx9k4_464555f7-8a36-4a47-8342-dfb9dbd5bae3/pull/0.log" Jul 13 22:46:18.238622 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:18.238601 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gx9k4_464555f7-8a36-4a47-8342-dfb9dbd5bae3/extract/0.log" Jul 13 22:46:18.359840 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:18.359805 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-64d6554466-4jltm_cfc6882c-943b-49f5-843d-490b9e6e075b/authorino/0.log" Jul 13 22:46:18.474815 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:18.474781 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-q5jsr_ed25e2c9-f71c-4537-a748-081bbb48b8fd/manager/0.log" Jul 13 22:46:18.588174 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:18.588089 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-kxv55_90b4731a-09ca-4b22-b625-65e0551117da/manager/0.log" Jul 13 22:46:18.819653 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:18.819619 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-85hq2_c832fc59-c341-4ff1-bd9c-193450f27e73/registry-server/0.log" Jul 13 22:46:18.939810 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:18.939776 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-rgbkf_f3d9361d-cb18-499e-9a90-484ea2d9f996/manager/0.log" Jul 13 22:46:19.169357 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:19.169328 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-6jqnv_2851b3be-0211-4fc2-85c0-b1dca7ea138c/manager/0.log" Jul 13 22:46:19.510931 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:19.510898 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf48rsjh_a0082143-c8bd-41a6-8632-3a816934bd8d/istio-proxy/0.log" Jul 13 22:46:19.839468 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:19.839389 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6bf5d9c687-q77l2_9247a6c2-1bd5-42a4-8a3a-b7c1f43bc10a/kube-auth-proxy/0.log" Jul 13 22:46:19.964379 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:19.964342 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-jhf2p_504e7d67-a82c-4653-bb50-a64a7a240eb5/istio-proxy/1.log" Jul 13 22:46:20.635460 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:20.635425 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-vgvcv_49c24049-acad-430c-8476-44c489ad4b32/main/0.log" Jul 13 22:46:25.391570 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:25.391534 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5tksb/must-gather-nvcsw"] Jul 13 22:46:25.391936 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:25.391907 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11e7e7a8-8f97-4fb4-beba-48299459a482" containerName="istio-proxy" Jul 13 22:46:25.391936 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:25.391920 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="11e7e7a8-8f97-4fb4-beba-48299459a482" containerName="istio-proxy" Jul 13 22:46:25.392016 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:25.391944 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fa0d14d-e36e-4479-b40b-364475e705ed" containerName="revoke-keys" Jul 13 22:46:25.392016 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:25.391949 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fa0d14d-e36e-4479-b40b-364475e705ed" containerName="revoke-keys" Jul 13 22:46:25.392016 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:25.391957 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6cfcba7-e72f-410a-9970-738567d220fb" containerName="istio-proxy" Jul 13 22:46:25.392016 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:25.391962 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6cfcba7-e72f-410a-9970-738567d220fb" containerName="istio-proxy" Jul 13 22:46:25.392016 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:25.391971 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6cfcba7-e72f-410a-9970-738567d220fb" containerName="istio-proxy" Jul 13 22:46:25.392016 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:25.391976 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6cfcba7-e72f-410a-9970-738567d220fb" containerName="istio-proxy" Jul 13 22:46:25.392305 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:25.392030 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="11e7e7a8-8f97-4fb4-beba-48299459a482" containerName="istio-proxy" Jul 13 22:46:25.392305 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:25.392036 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6cfcba7-e72f-410a-9970-738567d220fb" containerName="istio-proxy" Jul 13 22:46:25.392305 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:25.392043 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fa0d14d-e36e-4479-b40b-364475e705ed" containerName="revoke-keys" Jul 13 22:46:25.392305 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:25.392112 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11e7e7a8-8f97-4fb4-beba-48299459a482" containerName="istio-proxy" Jul 13 22:46:25.392305 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:25.392118 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="11e7e7a8-8f97-4fb4-beba-48299459a482" containerName="istio-proxy" Jul 13 22:46:25.392305 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:25.392191 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="11e7e7a8-8f97-4fb4-beba-48299459a482" containerName="istio-proxy" Jul 13 22:46:25.392305 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:25.392225 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6cfcba7-e72f-410a-9970-738567d220fb" containerName="istio-proxy" Jul 13 22:46:25.396934 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:25.396916 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5tksb/must-gather-nvcsw" Jul 13 22:46:25.400403 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:25.400366 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5tksb\"/\"kube-root-ca.crt\"" Jul 13 22:46:25.401075 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:25.401041 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5tksb\"/\"openshift-service-ca.crt\"" Jul 13 22:46:25.401257 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:25.401091 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5tksb\"/\"default-dockercfg-b7v8x\"" Jul 13 22:46:25.403993 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:25.403969 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5tksb/must-gather-nvcsw"] Jul 13 22:46:25.460043 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:25.460000 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/244448f1-a196-4ad0-b657-ae3e0fb6eb0f-must-gather-output\") pod \"must-gather-nvcsw\" (UID: \"244448f1-a196-4ad0-b657-ae3e0fb6eb0f\") " pod="openshift-must-gather-5tksb/must-gather-nvcsw" Jul 13 22:46:25.460043 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:25.460039 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwd5j\" (UniqueName: \"kubernetes.io/projected/244448f1-a196-4ad0-b657-ae3e0fb6eb0f-kube-api-access-pwd5j\") pod \"must-gather-nvcsw\" (UID: \"244448f1-a196-4ad0-b657-ae3e0fb6eb0f\") " pod="openshift-must-gather-5tksb/must-gather-nvcsw" Jul 13 22:46:25.561591 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:25.561551 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/244448f1-a196-4ad0-b657-ae3e0fb6eb0f-must-gather-output\") pod \"must-gather-nvcsw\" (UID: \"244448f1-a196-4ad0-b657-ae3e0fb6eb0f\") " pod="openshift-must-gather-5tksb/must-gather-nvcsw" Jul 13 22:46:25.561591 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:25.561596 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwd5j\" (UniqueName: \"kubernetes.io/projected/244448f1-a196-4ad0-b657-ae3e0fb6eb0f-kube-api-access-pwd5j\") pod \"must-gather-nvcsw\" (UID: \"244448f1-a196-4ad0-b657-ae3e0fb6eb0f\") " pod="openshift-must-gather-5tksb/must-gather-nvcsw" Jul 13 22:46:25.561867 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:25.561844 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/244448f1-a196-4ad0-b657-ae3e0fb6eb0f-must-gather-output\") pod \"must-gather-nvcsw\" (UID: \"244448f1-a196-4ad0-b657-ae3e0fb6eb0f\") " pod="openshift-must-gather-5tksb/must-gather-nvcsw" Jul 13 22:46:25.572474 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:25.572447 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwd5j\" (UniqueName: \"kubernetes.io/projected/244448f1-a196-4ad0-b657-ae3e0fb6eb0f-kube-api-access-pwd5j\") pod \"must-gather-nvcsw\" (UID: \"244448f1-a196-4ad0-b657-ae3e0fb6eb0f\") " pod="openshift-must-gather-5tksb/must-gather-nvcsw" Jul 13 22:46:25.706996 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:25.706893 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5tksb/must-gather-nvcsw" Jul 13 22:46:25.861912 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:25.861886 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5tksb/must-gather-nvcsw"] Jul 13 22:46:25.862506 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:46:25.862476 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod244448f1_a196_4ad0_b657_ae3e0fb6eb0f.slice/crio-adc4fcc9ff6faaf0f48fb8505dee3fa36e4712dd01e9d18013293e9878b24a37 WatchSource:0}: Error finding container adc4fcc9ff6faaf0f48fb8505dee3fa36e4712dd01e9d18013293e9878b24a37: Status 404 returned error can't find the container with id adc4fcc9ff6faaf0f48fb8505dee3fa36e4712dd01e9d18013293e9878b24a37 Jul 13 22:46:26.121660 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:26.121560 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5tksb/must-gather-nvcsw" event={"ID":"244448f1-a196-4ad0-b657-ae3e0fb6eb0f","Type":"ContainerStarted","Data":"adc4fcc9ff6faaf0f48fb8505dee3fa36e4712dd01e9d18013293e9878b24a37"} Jul 13 22:46:27.133936 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:27.133841 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5tksb/must-gather-nvcsw" event={"ID":"244448f1-a196-4ad0-b657-ae3e0fb6eb0f","Type":"ContainerStarted","Data":"f33ebe001a05e64085971a66fb2f450e5cf9b57f0119546e6abfd4b6ddce3b71"} Jul 13 22:46:27.133936 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:27.133903 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5tksb/must-gather-nvcsw" event={"ID":"244448f1-a196-4ad0-b657-ae3e0fb6eb0f","Type":"ContainerStarted","Data":"bb6828c244cb6b2d1a2b65638c9fffebb1000d1b6a98f4c8b3786ace061041b8"} Jul 13 22:46:27.157365 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:27.157296 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5tksb/must-gather-nvcsw" podStartSLOduration=1.302070434 podStartE2EDuration="2.157272598s" totalImagesPullingTime="855.202164ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:46:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:46:27.155646277 +0000 UTC m=+2793.149400569" watchObservedRunningTime="2026-07-13 22:46:27.157272598 +0000 UTC m=+2793.151026892" Jul 13 22:46:28.319648 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:28.319608 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5bkm9_56086743-e831-415a-a2c3-6982ef099751/global-pull-secret-syncer/0.log" Jul 13 22:46:28.469891 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:28.469850 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-q8965_061cea64-4ced-48b8-bd3a-3c6c22fc9e77/konnectivity-agent/0.log" Jul 13 22:46:28.511424 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:28.511388 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-234.ec2.internal_2df761e96899302463c4ca9d4dd2ee01/haproxy/0.log" Jul 13 22:46:33.399415 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:33.399381 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k5bfz_6e63fc16-f858-4f3c-84e5-3f89699f7250/extract/0.log" Jul 13 22:46:33.427784 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:33.427749 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k5bfz_6e63fc16-f858-4f3c-84e5-3f89699f7250/util/0.log" Jul 13 22:46:33.456987 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:33.456934 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759k5bfz_6e63fc16-f858-4f3c-84e5-3f89699f7250/pull/0.log" Jul 13 22:46:33.480735 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:33.480700 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xxfwl_6803f4ff-7912-492d-8540-6bc0c47650dd/extract/0.log" Jul 13 22:46:33.500589 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:33.500560 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xxfwl_6803f4ff-7912-492d-8540-6bc0c47650dd/util/0.log" Jul 13 22:46:33.526310 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:33.526231 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xxfwl_6803f4ff-7912-492d-8540-6bc0c47650dd/pull/0.log" Jul 13 22:46:33.551299 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:33.551263 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736cmcr_4387c748-a3eb-44ee-9016-2688537cf27f/extract/0.log" Jul 13 22:46:33.575028 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:33.574994 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736cmcr_4387c748-a3eb-44ee-9016-2688537cf27f/util/0.log" Jul 13 22:46:33.594905 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:33.594879 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736cmcr_4387c748-a3eb-44ee-9016-2688537cf27f/pull/0.log" Jul 13 22:46:33.644237 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:33.644188 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gx9k4_464555f7-8a36-4a47-8342-dfb9dbd5bae3/extract/0.log" Jul 13 22:46:33.665263 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:33.665162 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gx9k4_464555f7-8a36-4a47-8342-dfb9dbd5bae3/util/0.log" Jul 13 22:46:33.690304 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:33.690191 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gx9k4_464555f7-8a36-4a47-8342-dfb9dbd5bae3/pull/0.log" Jul 13 22:46:34.220556 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:34.220465 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-64d6554466-4jltm_cfc6882c-943b-49f5-843d-490b9e6e075b/authorino/0.log" Jul 13 22:46:34.307747 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:34.307708 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-q5jsr_ed25e2c9-f71c-4537-a748-081bbb48b8fd/manager/0.log" Jul 13 22:46:34.392493 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:34.392458 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-kxv55_90b4731a-09ca-4b22-b625-65e0551117da/manager/0.log" Jul 13 22:46:34.510357 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:34.510268 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-85hq2_c832fc59-c341-4ff1-bd9c-193450f27e73/registry-server/0.log" Jul 13 22:46:34.675010 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:34.674794 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-rgbkf_f3d9361d-cb18-499e-9a90-484ea2d9f996/manager/0.log" Jul 13 22:46:34.824234 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:34.824115 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-6jqnv_2851b3be-0211-4fc2-85c0-b1dca7ea138c/manager/0.log" Jul 13 22:46:36.648216 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:36.648165 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6ddff77d76-7764x_b9b7c5ca-fc37-4ffc-978d-a555355b02be/metrics-server/0.log" Jul 13 22:46:36.696710 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:36.696671 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5klkq_3982d16e-b33e-4fc4-b834-bd8f9b6a35ce/node-exporter/0.log" Jul 13 22:46:36.713875 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:36.713845 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5klkq_3982d16e-b33e-4fc4-b834-bd8f9b6a35ce/kube-rbac-proxy/0.log" Jul 13 22:46:36.753871 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:36.753845 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5klkq_3982d16e-b33e-4fc4-b834-bd8f9b6a35ce/init-textfile/0.log" Jul 13 22:46:37.354180 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:37.354145 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b49b6899d-nxgfv_f4ff6c51-71c0-4f77-a4ef-ef5b602360fc/thanos-query/0.log" Jul 13 22:46:37.371156 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:37.371131 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b49b6899d-nxgfv_f4ff6c51-71c0-4f77-a4ef-ef5b602360fc/kube-rbac-proxy-web/0.log" Jul 13 22:46:37.389470 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:37.389441 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b49b6899d-nxgfv_f4ff6c51-71c0-4f77-a4ef-ef5b602360fc/kube-rbac-proxy/0.log" Jul 13 22:46:37.407389 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:37.407359 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b49b6899d-nxgfv_f4ff6c51-71c0-4f77-a4ef-ef5b602360fc/prom-label-proxy/0.log" Jul 13 22:46:37.425642 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:37.425611 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b49b6899d-nxgfv_f4ff6c51-71c0-4f77-a4ef-ef5b602360fc/kube-rbac-proxy-rules/0.log" Jul 13 22:46:37.443033 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:37.443008 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b49b6899d-nxgfv_f4ff6c51-71c0-4f77-a4ef-ef5b602360fc/kube-rbac-proxy-metrics/0.log" Jul 13 22:46:38.261368 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:38.261331 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5tksb/perf-node-gather-daemonset-jdgkg"] Jul 13 22:46:38.269402 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:38.269366 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5tksb/perf-node-gather-daemonset-jdgkg" Jul 13 22:46:38.284435 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:38.284397 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5tksb/perf-node-gather-daemonset-jdgkg"] Jul 13 22:46:38.410573 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:38.410538 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5972575e-b170-49ec-9c61-82ec072e6b0c-podres\") pod \"perf-node-gather-daemonset-jdgkg\" (UID: \"5972575e-b170-49ec-9c61-82ec072e6b0c\") " pod="openshift-must-gather-5tksb/perf-node-gather-daemonset-jdgkg" Jul 13 22:46:38.410573 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:38.410584 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5972575e-b170-49ec-9c61-82ec072e6b0c-sys\") pod \"perf-node-gather-daemonset-jdgkg\" (UID: \"5972575e-b170-49ec-9c61-82ec072e6b0c\") " pod="openshift-must-gather-5tksb/perf-node-gather-daemonset-jdgkg" Jul 13 22:46:38.410857 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:38.410611 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5972575e-b170-49ec-9c61-82ec072e6b0c-proc\") pod \"perf-node-gather-daemonset-jdgkg\" (UID: \"5972575e-b170-49ec-9c61-82ec072e6b0c\") " pod="openshift-must-gather-5tksb/perf-node-gather-daemonset-jdgkg" Jul 13 22:46:38.410857 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:38.410771 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d2vn\" (UniqueName: \"kubernetes.io/projected/5972575e-b170-49ec-9c61-82ec072e6b0c-kube-api-access-5d2vn\") pod \"perf-node-gather-daemonset-jdgkg\" (UID: \"5972575e-b170-49ec-9c61-82ec072e6b0c\") " pod="openshift-must-gather-5tksb/perf-node-gather-daemonset-jdgkg" Jul 13 22:46:38.410857 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:38.410842 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5972575e-b170-49ec-9c61-82ec072e6b0c-lib-modules\") pod \"perf-node-gather-daemonset-jdgkg\" (UID: \"5972575e-b170-49ec-9c61-82ec072e6b0c\") " pod="openshift-must-gather-5tksb/perf-node-gather-daemonset-jdgkg" Jul 13 22:46:38.511831 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:38.511722 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5972575e-b170-49ec-9c61-82ec072e6b0c-lib-modules\") pod \"perf-node-gather-daemonset-jdgkg\" (UID: \"5972575e-b170-49ec-9c61-82ec072e6b0c\") " pod="openshift-must-gather-5tksb/perf-node-gather-daemonset-jdgkg" Jul 13 22:46:38.512026 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:38.511830 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5972575e-b170-49ec-9c61-82ec072e6b0c-podres\") pod \"perf-node-gather-daemonset-jdgkg\" (UID: \"5972575e-b170-49ec-9c61-82ec072e6b0c\") " pod="openshift-must-gather-5tksb/perf-node-gather-daemonset-jdgkg" Jul 13 22:46:38.512026 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:38.511869 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5972575e-b170-49ec-9c61-82ec072e6b0c-sys\") pod \"perf-node-gather-daemonset-jdgkg\" (UID: \"5972575e-b170-49ec-9c61-82ec072e6b0c\") " pod="openshift-must-gather-5tksb/perf-node-gather-daemonset-jdgkg" Jul 13 22:46:38.512026 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:38.511897 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5972575e-b170-49ec-9c61-82ec072e6b0c-proc\") pod \"perf-node-gather-daemonset-jdgkg\" (UID: \"5972575e-b170-49ec-9c61-82ec072e6b0c\") " pod="openshift-must-gather-5tksb/perf-node-gather-daemonset-jdgkg" Jul 13 22:46:38.512026 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:38.511957 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d2vn\" (UniqueName: \"kubernetes.io/projected/5972575e-b170-49ec-9c61-82ec072e6b0c-kube-api-access-5d2vn\") pod \"perf-node-gather-daemonset-jdgkg\" (UID: \"5972575e-b170-49ec-9c61-82ec072e6b0c\") " pod="openshift-must-gather-5tksb/perf-node-gather-daemonset-jdgkg" Jul 13 22:46:38.512520 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:38.512403 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5972575e-b170-49ec-9c61-82ec072e6b0c-sys\") pod \"perf-node-gather-daemonset-jdgkg\" (UID: \"5972575e-b170-49ec-9c61-82ec072e6b0c\") " pod="openshift-must-gather-5tksb/perf-node-gather-daemonset-jdgkg" Jul 13 22:46:38.512520 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:38.512431 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5972575e-b170-49ec-9c61-82ec072e6b0c-proc\") pod \"perf-node-gather-daemonset-jdgkg\" (UID: \"5972575e-b170-49ec-9c61-82ec072e6b0c\") " pod="openshift-must-gather-5tksb/perf-node-gather-daemonset-jdgkg" Jul 13 22:46:38.512520 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:38.512471 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5972575e-b170-49ec-9c61-82ec072e6b0c-lib-modules\") pod \"perf-node-gather-daemonset-jdgkg\" (UID: \"5972575e-b170-49ec-9c61-82ec072e6b0c\") " pod="openshift-must-gather-5tksb/perf-node-gather-daemonset-jdgkg" Jul 13 22:46:38.512520 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:38.512492 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5972575e-b170-49ec-9c61-82ec072e6b0c-podres\") pod \"perf-node-gather-daemonset-jdgkg\" (UID: \"5972575e-b170-49ec-9c61-82ec072e6b0c\") " pod="openshift-must-gather-5tksb/perf-node-gather-daemonset-jdgkg" Jul 13 22:46:38.531036 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:38.530999 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d2vn\" (UniqueName: \"kubernetes.io/projected/5972575e-b170-49ec-9c61-82ec072e6b0c-kube-api-access-5d2vn\") pod \"perf-node-gather-daemonset-jdgkg\" (UID: \"5972575e-b170-49ec-9c61-82ec072e6b0c\") " pod="openshift-must-gather-5tksb/perf-node-gather-daemonset-jdgkg" Jul 13 22:46:38.584096 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:38.584062 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5tksb/perf-node-gather-daemonset-jdgkg" Jul 13 22:46:38.781881 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:38.781853 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5tksb/perf-node-gather-daemonset-jdgkg"] Jul 13 22:46:38.783391 ip-10-0-133-234 kubenswrapper[2555]: W0713 22:46:38.783309 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5972575e_b170_49ec_9c61_82ec072e6b0c.slice/crio-04244058b780cd332c63213db773e24b8b9097e811c41a06271ac9ef2a8a7f1e WatchSource:0}: Error finding container 04244058b780cd332c63213db773e24b8b9097e811c41a06271ac9ef2a8a7f1e: Status 404 returned error can't find the container with id 04244058b780cd332c63213db773e24b8b9097e811c41a06271ac9ef2a8a7f1e Jul 13 22:46:39.279534 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:39.278698 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5tksb/perf-node-gather-daemonset-jdgkg" event={"ID":"5972575e-b170-49ec-9c61-82ec072e6b0c","Type":"ContainerStarted","Data":"7508edd13b5f96ec7b1613f117af75fb1cb6afb2b95c51c5749bfa985f025b48"} Jul 13 22:46:39.279534 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:39.278795 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5tksb/perf-node-gather-daemonset-jdgkg" event={"ID":"5972575e-b170-49ec-9c61-82ec072e6b0c","Type":"ContainerStarted","Data":"04244058b780cd332c63213db773e24b8b9097e811c41a06271ac9ef2a8a7f1e"} Jul 13 22:46:39.280072 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:39.279592 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5tksb/perf-node-gather-daemonset-jdgkg" Jul 13 22:46:39.296916 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:39.295431 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5tksb/perf-node-gather-daemonset-jdgkg" podStartSLOduration=1.295413756 podStartE2EDuration="1.295413756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 22:46:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:46:39.295089764 +0000 UTC m=+2805.288844077" watchObservedRunningTime="2026-07-13 22:46:39.295413756 +0000 UTC m=+2805.289168049" Jul 13 22:46:41.251438 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:41.251404 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pqjkz_54f54657-10eb-47ae-ba8f-ec453e686d04/dns/0.log" Jul 13 22:46:41.268068 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:41.268040 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pqjkz_54f54657-10eb-47ae-ba8f-ec453e686d04/kube-rbac-proxy/0.log" Jul 13 22:46:41.372181 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:41.372135 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2znr7_9985a1e9-5e38-4bea-9745-988c4524fc21/dns-node-resolver/0.log" Jul 13 22:46:41.976882 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:41.976840 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6nq95_54550ef2-07f8-4e03-b898-c5d7536cc61e/node-ca/0.log" Jul 13 22:46:42.906766 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:42.906724 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf48rsjh_a0082143-c8bd-41a6-8632-3a816934bd8d/istio-proxy/0.log" Jul 13 22:46:43.055796 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:43.055767 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6bf5d9c687-q77l2_9247a6c2-1bd5-42a4-8a3a-b7c1f43bc10a/kube-auth-proxy/0.log" Jul 13 22:46:43.084424 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:43.084358 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-jhf2p_504e7d67-a82c-4653-bb50-a64a7a240eb5/istio-proxy/0.log" Jul 13 22:46:43.146603 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:43.146569 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-jhf2p_504e7d67-a82c-4653-bb50-a64a7a240eb5/istio-proxy/1.log" Jul 13 22:46:43.923562 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:43.923525 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7hh44_42289584-c895-4ac8-a13f-d718fa245599/serve-healthcheck-canary/0.log" Jul 13 22:46:44.426426 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:44.426391 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kr6fz_9d5db581-bfb8-4fa9-a027-80f98597e1b0/kube-rbac-proxy/0.log" Jul 13 22:46:44.458379 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:44.458345 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kr6fz_9d5db581-bfb8-4fa9-a027-80f98597e1b0/exporter/0.log" Jul 13 22:46:44.487168 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:44.487139 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kr6fz_9d5db581-bfb8-4fa9-a027-80f98597e1b0/extractor/0.log" Jul 13 22:46:46.310393 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:46.310361 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5tksb/perf-node-gather-daemonset-jdgkg" Jul 13 22:46:47.427591 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:47.427566 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-5mxzw_a5adea24-b5bc-4fc6-bedd-9357092f0205/manager/0.log" Jul 13 22:46:47.685624 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:47.685565 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-5k88d_dba67c81-79d8-4b45-8382-e7f9bfe110dc/manager/0.log" Jul 13 22:46:47.811382 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:47.811352 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-xmzbx_77499f45-4453-4263-aa49-e3a9a534e134/server/0.log" Jul 13 22:46:47.915375 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:47.915337 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-rn2vl_4035fc64-bb79-42c7-9d2c-a95f42177716/manager/0.log" Jul 13 22:46:47.938552 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:47.938472 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67ddb65fc-dwcsd_2f424140-dd4b-4331-8e78-cd10ee0f70e2/manager/0.log" Jul 13 22:46:47.955419 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:47.955384 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67ddb65fc-dwcsd_2f424140-dd4b-4331-8e78-cd10ee0f70e2/copy-manifests/0.log" Jul 13 22:46:48.886401 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:48.886367 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf48rsjh_a0082143-c8bd-41a6-8632-3a816934bd8d/istio-proxy/0.log" Jul 13 22:46:49.021580 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:49.021547 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6bf5d9c687-q77l2_9247a6c2-1bd5-42a4-8a3a-b7c1f43bc10a/kube-auth-proxy/0.log" Jul 13 22:46:49.040182 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:49.040145 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-jhf2p_504e7d67-a82c-4653-bb50-a64a7a240eb5/istio-proxy/0.log" Jul 13 22:46:49.108451 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:49.105885 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-jhf2p_504e7d67-a82c-4653-bb50-a64a7a240eb5/istio-proxy/1.log" Jul 13 22:46:49.462667 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:49.462627 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-blmfj_1b01374e-3cab-4b97-9537-73535f0fd133/openshift-lws-operator/0.log" Jul 13 22:46:50.380650 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:50.380608 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf48rsjh_a0082143-c8bd-41a6-8632-3a816934bd8d/istio-proxy/0.log" Jul 13 22:46:50.525070 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:50.525039 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6bf5d9c687-q77l2_9247a6c2-1bd5-42a4-8a3a-b7c1f43bc10a/kube-auth-proxy/0.log" Jul 13 22:46:50.548125 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:50.548090 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-jhf2p_504e7d67-a82c-4653-bb50-a64a7a240eb5/istio-proxy/0.log" Jul 13 22:46:50.615320 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:50.615222 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-jhf2p_504e7d67-a82c-4653-bb50-a64a7a240eb5/istio-proxy/1.log" Jul 13 22:46:55.590018 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:55.589940 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-86hvh_22a82d33-c9c8-445a-a814-5534be9dd95e/kube-multus/0.log" Jul 13 22:46:55.613270 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:55.613240 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6xnxl_d94f40ee-a429-4bf2-b7cb-c436922cdedc/kube-multus-additional-cni-plugins/0.log" Jul 13 22:46:55.631288 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:55.631259 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6xnxl_d94f40ee-a429-4bf2-b7cb-c436922cdedc/egress-router-binary-copy/0.log" Jul 13 22:46:55.649577 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:55.649558 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6xnxl_d94f40ee-a429-4bf2-b7cb-c436922cdedc/cni-plugins/0.log" Jul 13 22:46:55.666225 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:55.666177 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6xnxl_d94f40ee-a429-4bf2-b7cb-c436922cdedc/bond-cni-plugin/0.log" Jul 13 22:46:55.682513 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:55.682498 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6xnxl_d94f40ee-a429-4bf2-b7cb-c436922cdedc/routeoverride-cni/0.log" Jul 13 22:46:55.705175 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:55.705152 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6xnxl_d94f40ee-a429-4bf2-b7cb-c436922cdedc/whereabouts-cni-bincopy/0.log" Jul 13 22:46:55.724239 ip-10-0-133-234 kubenswrapper[2555]: I0713 22:46:55.724193 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6xnxl_d94f40ee-a429-4bf2-b7cb-c436922cdedc/whereabouts-cni/0.log"